Microsoft Fabric

Anyone using Microsoft Fabric at work (or for hobby if you can afford it)?

If so, how do you find it, you reckon Fabric will catch up to Databricks and become the next big thing in the cloud analytics platform like how Power BI slowly became the king in Visualisation space?

 

We use Databricks, and I've had a few interactions with our Microsoft reps with the usual demos / workshops when it first was announced. So we don't have it rolled out but have had a few conversations about how it might fit within our ecosystem. Mostly for for data integration and engineering, and have some analyst-types doing statistical work on the platform as well. There is limited use of the "analytics" features save for some dashboards wrapping a self serve data check-type purpose. Most of the regular enterprise reporting we have is with Power BI, and even though there are limitations, the Databricks "dashboarding" experience is still not sufficiently mature, and for those that need proper control they can work with R or python to put together figures. It feels like the focus is currently on rolling out Genie for more of a chat / AI interface to your data instead

Some initial thoughts:
- The capacity model for Fabric has positives and negatives - from a business unit point of view it will mean we worry less about consumption. It is very functional, and we can access capacity quite readily to run all sorts of workloads, with isolation well implemented from ground up. I've heard they've taken a different approach where workloads are being throttled rather than capacity being scaled as with current Power BI Premium capacity? A few of us have worries about resource contention especially if we can't prioritise workloads. I.e. a poorly written analytics workload prevents the critical workload from finishing on time
- I'm not sure what the go is with the really sensitive workloads nowadays - with Fabric being all capacity based and Databricks pushing serverless. You can't really do the usual self-hosted runtime thing you used to be able to do with ADF; with Databricks at least you can make it use a dedicated compute that is within your tenancy
- The proprietary secret sauce delta lake optimisations that are being touted are distasteful. Classic Microsoft thing to do though. Same with compatibility with things like Common Data Model format data - it would be well supported (in a don't have to know why it works but it works so I guess that's ok type of way) in Microsoft products with the open driver on github being like 5 versions behind with serious issues like incremental loads still not being supported?
- Purview integration still wasn't there when we looked at it, and end-to-end data governance feels more difficult to get a handle of. Might be more my inexperience with the product. Unity catalog feels more mature and production ready, not that it's perfect either, but certainly has enough to get things done.
- They've brought all the different Azure tools together, but it still feels like all the different bits and pieces are still spread out everywhere? Again may be due to my inexperience. I also hate the way some of the previous tools used to do things. The lack of version control for Azure Machine Learning, the terrible way it was implemented for Power BI, the dumbed down Azure Data Factory "actions". Nowadays scripting up a spark call is super consistent using AI now so the benefits of having a gui to do that (in terms of making it quicker and more accessible) are really debatable.
- Databricks autoloader is great so unless Fabric can come up with something similar that's a big gap. Automatic handling of incremental / change data, schema evolution, constraints (with the ability to rescue data), handling and mapping complex data types
- Multi-cloud would probably be better supported with Databricks with the delta sharing stuff

That being said, both systems do fairly similar things so you could probably get anything you can do in one in the other. Also consider access to support - I've had a really spotty experience with Microsoft support, if you develop a relationship with the MS solution architects/engineers who are supporting your contract and have them help you with things that works well. The ticket system is horrible though (all parts of it including navigating the ticket submission process all the way to trying to communicate with the offshore support), even for serious issues in Microsoft's court. On the Databricks side the senior solutions architect supporting our sector is really good - he gives out his mobile and he responds to a Teams message usually within a few minutes, you can ask anything from architecture to pyspark to putting together a poc for a requirement, and if he doesn't know right away he'll go pull together the info for you or bring in a specialist resource. Like having direct access to level 2/3 support.

 

FYI - learn Apache Spark - https://sparkui.fabric.microsoft.com/sparkui user view https://app.fabric.microsoft.com/workloads/data-pipeline/monitoring



Most of fabric pipeline is a fancy frontend with spark doing the heavy lifting.

 

damn that is detailed reply.

On Thursday i'll be able to use Azure ADF, Synapse, Azure Serverless storage and Fabric and learn from a contractor. Very excited.

what do you think of this cert
https://www.databricks.com/learn/certification/apache-spark-developer-associate