BW is Dead – Long Live BW

Today we lay BW to rest and celebrate its 16 years on this earth as the first 3^rd generation packaged data warehouse. It changed how we built reporting systems; it created the idea of standard content; and it was one of the most successful platforms launched by any software company.

In 1998 Dr. Berg was asked to come to Stockholm by a global telecom to head up their implementation of a new tool called Business Information Warehouse (BIW). This was a new tool set with pre-developed extract, transform, and load (ETL) programs, had a model-driven approach to EDWs, and it was just released as a 1.0E version.

Over the next 16 years we would watch the child grow up, marry and raise children of its own. On March 24, 2014, the first generation of this family passed away₍₁₎, but we live with its legacy and we can see a great potential in the next generation of EDWs and its grandchildren. In this obituary we celebrate BW’s long life with us and look to the future.

The Infant Years (1998-2000)

When BW was born it had several pre-delivered operational data stores (ODS); a star schema data model called InfoCubes and an interface to Excel called Business Explorer (BEx). However, it also had a lot of infant issues and challenges. In 1998 it was formally released as 1.2a and later as a production release called 1.2b. In this time period the ODS was not really reportable, the front-end interface to other reporting tools did not really work, and the data models were complex. A lot of changes laid ahead.

The Childhood Years (2000-2001)

Even as a toddler, BW changed dramatically. In version 2.0a the ODS was renamed, to what it really was, a “persistent staging area” (PSA), and a new true ODS was introduced. The star schema model was simplified and the interface to other front-end tool started to really work. The changes were dramatic. With version 2.0b the toddler could now walk and version 2.1c gave it new clothes in form of its first really usable standard content.

Many criticized the child and failed to see the potential in the kid who would grow up and change how we view data warehouses forever. In these childhood years, it got its own user conferences and was adopted by parents all over the world. The expectations were high.

The Teenager Years (2002-2003)

With version 3.0a BW started to play nice with other kids. A new charting engine and web service allowed web applications to be built. The new DB Link allowed data to be accessed from non-SAP sources, XML was supported, and common warehouse metadata (CWM) was also added. Internally, the teenager could now see errors in transfer and update rules, and you could report directly on the ODSs. Formula editors and the Archiving Development Kit (ADK) were also included. The teenager was becoming an adult.

With version 3.0b and later 3.1C, BW went from a pimply teenager to a grownup hungry for more knowledge. Sure, there were features missing such as internal process chains for data loading scheduling and broadcasting. But these would come later and over 3,000 parents believed in the teenager and had adopted him.

The Adult Years (2004-2006)

Skipping a class (version 3.4), BW entered the adult years in its 3.5 release in 2004. It now supported Business Planning and Simulation (BPS) on its shoulders; it had a real set of working scheduling tools; it supported broadcasting of reports; it was now part of NetWeaver; and it got a knowledge manager to do content ratings, searches, feedback, subscriptions and discussions. The Web Application Designer was enhanced so that corporate web templates could be reused and the Analysis Process Designer (APD) was made truly useful by adding classifications, stratification and pre-processing for data mining. In addition, Universal Data Connect and Java connections were added to the Open Hub data movement platform.

On the front end a Microsoft BI OLE connector and a BI XMLA connector for web based OLAP sources was added. Unicode was supported for the first time and queries could now be published to the SAP Portal Content Catalog. iViews was launched and knowledge management object to a collaboration room on the web portal was enabled. With the late release of NW 2004s it broke with its version naming conventions but would later come back to its senses and continue a more conventional approach to version names.

In this mature period, BW was living in its house on NetWeaver Street and was now a productive corporate citizen with an increasing number of adaptations worldwide. America no longer viewed it as an unruly teenager and Asia finally stopped kicking its legs to see how it would react. BW was earning respect.

The Middle-Aged Years (2006-2009)

BW was now a productive member of society striving for perfection. In version 2004s (released unfortunately in 2006), it got a new much needed BEx Query Designer; it tried a new tool called BEx Report Designer that never really took off. It cleaned its older data models and added a new remodeling toolbox. It changed the locks on the house and introduced new analysis authorization security. BW got a graphical interface for ETL and integrated planning was launched. Real-time data loads were added and BEx Analyzer became much more interactive.

BW also dumped some of its old friends like BEx Browser and the Ad-hoc Query Designer. Java became its new hang-out buddy. BW Accelerator (BWA) was its middle-age crisis sport car for fast fun, and the administrative cockpit was now used for central management of the many assets accumulated over a long life. In the garage the Administrator Workbench was replaced with a new larger version called Data Warehousing Workbench.

Its old mistress, Web Application Designer got a facelift and BEx Broadcaster was enhanced in a health studio. Even its old data movement was simplified into a new data transfer process (DTP) and write-optimized DSOs were introduced to help with slow performance. With the release of version 7.0, BW was entering his best and most mature years.

Old Age (2010-2013)

BW started to show its age in 2010. The data volumes were no longer small. Data had grown from a few Gigabytes in 1998 to many Terabytes by 2010 and BW was struggling to keep up. Workarounds existed in form of external indexing engines like BWA for faster query performance and a new Layered Scalable Architecture (LSA) for partitioning the data. Write-Optimized DSOs were used to get faster data loading, but it was all too obvious, BW was getting into old age.

Increased pressure was mounting for faster mobile applications and BW teamed up with an old friend called Sybase to get some new capabilities a few years before. The front end was getting stale, but a new family member who joined in 2008, called BusinessObjects took care of that with new and cool front-end tools. Its longtime colleague, Visual Composer, passed away quietly one night and BW was getting into its ripe old age.

Meanwhile, its children were doing extremely well. Its stepdaughter, HANA was born in 2009 and was a fast sprinter with awesome potential. Its son, BusinessObjects had children of its own, called Design Studio and Analysis. Its other step-son, Sybase, had also a son called IQ Near-Line Storage that helped BW store some old data in its closet where it could be easily accessed when needed. It was now clear that BW would live on through its children and grand-children.

With release 7.3 BW partially moved into its new HANA house. It still hung out with its old RDBMS friends, but was now looking to a new world of in-memory living accommodations. New performance monitors were supplemented and Semantically Partitioned Objects (SPO) were added to keep the lights on for older databases. Hybrid providers were created to link to BWA for performance, or to ECC for real-time data. Data activation was made faster by package fetches, data flow wizards were added for faster development and LSA templates were included to keep data stores smaller. BWA was asked to do more of the heavy lifting for BW with modeling and some OLAP functionality being moved to it in version 7.2.

Passing Away (2014)

After a long and productive life, BW passed away quietly in its sleep in Orlando, Florida at the BI and HANA conference on March 24, 2014. It will be missed. It changed our world but its children will be with us for years to come.

A New Beginning with BW 7.4 (2014)

With the birth of its last child, BW 7.4, born on the HANA in-memory database platform, BW will live on. This version is the first optimized to take full advantage of the new technology.

It is up to 900 times faster, it has smaller footprint with up to 10 times compression of data. It does not need PSAs; it has new data load capabilities such as Smart Data Access (SDA) Operational Data Provisioning (ODP) and Operational Delta Queues (ODQ). Furthermore, InfoCubes are only needed in special circumstances such as write-backs and non-cumulative key figures, and the data architecture is much simpler.

Data activations and transformations are now done in-memory and unified modeling is done in Eclipse modeling tools. In addition, BW and non-BW HANA models can now be queried in Open ODS views and through enhanced composite providers. New HANA-specific content is available in content release 7.47 (cool name), DSO optimization for HANA is not required, virtual data models can be created, the OLAP cache is rewritten, DBA cockpit is enhanced, HANA Analysis Processes (HAP) is added, and many hardware vendors have now stopped selling BWA. So the new BW is really a changed citizen of the new in-memory world.

Do We Need BW 7.4?

Some have pointed out that as the SAP Business Suite moved to HANA in January 2013 and HANA Live was released with its 650+ views for reporting last fall, we can do almost all operational reporting there and do not need BW. That is a naïve and simplistic view.

First, we need a platform for merging data from a variety of sources. Second, we need a memory layer for corporate reporting. If we don’t have that, we would be unable to recreate reports from prior periods when we changed CCA, PCA and organizational hierarchies; or when we change chart-of accounts in the general ledger. Third, we are likely to see an explosion in data volumes from non-traditional sources and unstructured data in the next decade. It would be both counter-intuitive and naïve to believe that we can manage this complexity and data volume on a single platform if we want to also have agility and flexibility.

Finally, the enterprise data warehouse (EDW) services required to provide timely, reliable, consistent information and insight are not provided by HANA or any other DBMS alone. They are provided by BW in a way that can be extended beyond the traditional monolithic EDW architecture to support the Logical Data Warehouse concept and architecture.

However, BW may now revert to its original intention of being a business intelligence and analytics platform for enterprise management, reporting and planning. It is also a platform to enterprise data warehouses for mobile analysis and it can now support tens of thousands of users.

In my own test of the new Ivy Bridge Intel processors for 2014, HANA is also 268% faster than those HANA systems implemented last year (15 cores vs. 10 per processors and faster processor speeds). So we are seeing the first leaps in technology instead of the incremental improvements we saw in the last decade.

So, are you going to revert to the old reporting ways, or do you understand the benefits and architecture of an EDW?  We are strong believers in the latter. But we cannot continue to build BW on HANA the same way we have over the last 16 years. PSA is dead, InfoCubes are dying, ETL has changed, data architectures are modified (LSA++) and BW on HANA is much simpler to develop. We have 100 percent support for the old models, but it is time to say goodbye to the old BW.

(1) BW on any of the previously supported platforms will continue to be supported according to SAP’s standard support policy

What are the ride situations?