Keep Calm and Carry On with Discovery Hub®

1. Will Discovery Hub® lock us into a “black box” solution?

The high degree of automation is considered a key selling point for many Discovery Hub® developers. Most businesses have experienced some sort of technology or vendor lock-in due to a high degree of complexity and almost no transparency under the hood. Some are concerned that Discovery Hub® is a “black box” with no visibility to the internal mechanics, leaving developers blindly guessing as to what’s going on.

Luckily, with Discovery Hub®, you can stay calm and enjoy the great automation capabilities with full transparency. Developers with Microsoft SQL Server experience can easily decipher what’s happening because:

• Discovery Hub® utilizes SQL Server products as the underlying engine that runs everything. All logical objects in Discovery Hub® (databases, tabular models, cubes, schemas, tables, views, stored procedures, SSIS Packages, etc.) are deployed physically to the SQL Server. You can simply review a solution using SQL Server Management Studio (SSMS).
• Inside Discovery Hub®, developers can easily access (and edit!) auto-generated scripts for data objects like transformation, cleansing, and data transfer processes. Additionally, deployment scripts for all objects can be retrieved from the deployment log in context and execution order.

Embrace automation and gain control and visibility into your BI and analytics tools. With Discovery Hub® it’s easy to “keep calm and carry on.”

2. Why would we need Discovery Hub® if we can prepare data directly in our visualization tool?

Several visualization tools on the BI market enable preparation of data ranging from simple data transformation and cleaning operations to multi-layered, persisted staging of data (enables advanced modeling). If we assume that the front-end provides the right analytics to the appropriate people in a timely fashion, then why would anybody need a Discovery Hub® platform between data sources and the front-end?

There are several reasons why managing all the business logic inside a front-end tool may not be a good idea. Let me explain why:

• An entire BI solution would need to be rebuilt if the business decides to switch visualization tools in the future. Switching costs can be significant, meaning your Finance Director might hyperventilate.
• It is very difficult to manage multiple visualization tools within an organization. An entire organization could be forced to use a single tool regardless of function and need. Progress may be delayed because all new developments need to be duplicated across every unique tool – each with their own merits, features, reporting capabilities and user communities. Each tool could become a silo, risking inconsistency for the scope of the data, data quality, and reporting capabilities.
• For many businesses, it’s a somewhat uncomfortable situation to place all business and reporting logic into front-end applications. Such logic often reflects decisions made at an executive level. It can be ethically or even legally sensitive and potentially limit the opportunity to utilize the full value of BI tools by opening it to data-deprived business users.

Having a Discovery Hub solution between data sources and front-ends will mitigate risks and provide a more future-proof and flexible BI solution. And this is a good thing.

3. What if Discovery Hub® is not able to automate everything?

Discovery Hub® aims to cover the gap between all data sources and front-ends by leveraging the power of automation. Discovery Hub® facilitates faster development cycles and automates many of those tedious tasks that drain our time and keep us from the rewarding projects that add real value. Obviously, the software can’t anticipate and support every possible scenario. If requirements dictate a solution outside the scope of what Discovery Hub® can automate, then what?

• Discovery Hub® includes custom code objects that are easy to integrate alongside the auto-generated code. These custom code objects enable developers to define operations and logic based on T-SQL script. Specifically:

– Custom Field Transformations
– Views
– Stored Procedure
– User Defined Functions
– Script Actions

• Alternatively, a developer can simply edit the auto-generated T-SQL scripts directly or replace with an SSIS package. The SSIS integration is practical as it allows developers to interact with external applications and run operations that are outside the scope of a normal BI platform. It’s also convenient if you already have SSIS packages running ETL when starting a Discovery Hub project, as they can easily be re-used and save time. Developers can customize T-SQL in SSMS and SSIS packages in SSDT.

Remember that Discovery Hub® is using Microsoft SQL Server as an engine. You can always extend a Discovery Hub® solution with the full capacity of SQL Server. Ideally, you want to avoid going completely outside Discovery Hub given that documentation and maintenance can become a pain, but there are no technical restrictions imposed.

4. Does Discovery Hub® work for large companies?

The physical constraints and scalability of Discovery Hub® are largely given by the capabilities of Microsoft SQL Server. Smaller companies may run everything on a single server, while larger companies may need to split the load into multiple servers or use databases in the cloud. Not a problem for Discovery Hub®!

Whether it works for large companies is more of a political discussion than a technical one. Large companies often have a data warehouse in place and may have created dreadfully agonizing processes and policies to maintain it. It might be completely hand-coded and run with jaw-breaking speed but simply lacks the agility to keep up with user demands. No doubt, that existing platform was probably expensive and the famous sunk-cost fallacy may also play a role, so you can see why an enterprise may be reluctant to stray too far from earlier decisions. Often Discovery Hub® is brought in to accelerate and complement the development of business intelligence capabilities alongside an existing platform. An enterprise might create divisional or departmental Discovery Hubs to enable easier and faster localized BI platforms while continuing to use an existing data warehouse at a corporate level. Some companies utilize the flexibility and speed of Discovery Hub® by placing it between their existing data warehouse, raw data sources, and the front-ends. In Discovery Hub®, ongoing changes and new experiments are created and validated more rapidly to fit user needs. Meanwhile, Discovery Hub® also serves as a “blueprint” for future improvements to the corporate data warehouse and saves ETL developers from the onslaught of change requests from business users.

Even when a large company decides to replace their existing platform with Discovery Hub®, the transition takes time so it’s important that the platforms can co-exist and seamlessly integrate. From a technical standpoint, Discovery Hub ®contains several features that enable a smooth transition and make the life for many much easier.

5. Is Discovery Hub® relevant if migrating to a new data source?

In some instances, it makes perfect sense to focus on implementing back-end systems first, before even considering front-end visualization. Obviously, all dashboards sourcing new data are empty until data is created. Following a system implementation, often there’s a rush of new users storming the new system, attempting to understand and absorb with little consideration for future BI solutions. Those shortsighted BI projects created in haste often reveal truly bad practices later down the road. At that point, it’s too late to address data quality. Running a Discovery Hub® project alongside implementation will not only ensure your reporting capabilities will be ready sooner, but it can also facilitate better use of the new system and likely improve data quality significantly.

The business logic, naming conventions, how to measure success, KPIs, should largely be independent of underlying data sources. By starting before a new system goes live means that the data model will be ready to incorporate the legacy system with the new system (if you’re a BI Specialist, you should be experiencing a great sigh of relief). And let’s not forget, there is typically more than one relevant data source for reporting and it is unlikely that all data sources will be replaced at the same time.

Developing on top of a dynamic source is only possible with the level of automation that Discovery Hub® enables – you would need to map the new tables into existing data flows. Some logic would most likely need to be adjusted given that each system is different. Fortunately, switching and adding new data sources to existing solutions is just one of the many benefits that Discovery Hub® offers. Discover the answers to key questions on Discovery Hub® here.

Jacob Ross-Andersen
Solutions Specialist, TimeXtender