Today (29th of November 2013) I have just collected from Amazon locker my new book Extreme Scoping which was written by Larissa T. Moss which is "An Agile Approach to Enterprise Data Warehousing and Business Intelligence".
Several days ago, it was our 5th Anniversary of setting up our BI Consulting and Training company so it's been a few years in BI all together and a few places (2/3 a year) and I must admit companies do struggle with BI/DW deliverables at the beginning but even more after 1st and 2nd year.
My solution? Well.. Always evolving, but so far I have used Scrum with adjustments (as some methods should not be used with BI/DW Projects), Kanban and TOGAF and it's been recently that during our Christmas BI Competition Dominic Adamczyk (who shares great SQL/BI articles every Friday via email) mentioned Extreme Scoping; which was not the correct answer :) but after short research I realized it might help me with finding a method purely for BI (Scrum isn't pure but was best of all methods I knew).
In this blog post I will do detailed review of the book and I hope I will get answers to the following questions:
- Is Extreme Scoping really better than Scrum (which isn't perfect for BI / DW)? and how does it, achieve it?
- What have I been missing or what have I not noticed or experienced yet?
- Is Extreme Scoping going to add some value to TOGAF (which is an Enterprise Architecture Framework)? (I doubt it will try to make TOGAF redundant as it most likely will be a very important part of BI/DW Projects).
- Most importantly, how can Extreme Scoping help me with BI projects? and it is very convenient time for me as I will be starting a new contract at Ofsted on 12th of December 2013 as a Data Systems Architect (+ Trainer and maybe Team Lead?) so I hope to gain new knowledge and get a better start from day one :)
This will be in-depth review and I split into different days and I hope this will also spark some discussions! as I am really excited about this book I will most likely finish it within 2 days :)
IMPORTANT: First several chapters try to discuss several books in several pages. If you haven't read several BI related books or you do not have a considerable amount of experience in BI than you may feel confused.... a lot! You might have to ignore it, but the good news is that you may not have to come back to it if you start using Extreme Scoping.
List of days:
I have opened the book and I must admit layout, font and structure make it very readable and there are plenty of diagrams and tables that are also very readable (good start!).
The author shares her experience with her first DW/BI project which sounds like a nightmare, but it is not too far away from a typical project. I admire the author for being able to cope for about a year! In my case when someone threatens me (fired, verbal abuse, etc.) which is rare, but happens then I give me resignation the next day! No point arguing with ....
Fortunately, companies, people and tools "matured" and some kind of agile approach is common place which makes the project run, maybe it is not the most effective, but when people cannot compare it with anything else then it is difficult to argue it is not how it should be; especially that many companies still seem to not bother too much about ROI (Return or Investment) and low TCO (Total Cost of Ownership) and just go with the flow.
Preface also mentioned XP (Extreme Programming) and Scrum and gave overview of the book which got me even more excited now that I experienced author direct and pleasant way of writing.
Chapter 1 tries to describe everything about BI/DW in the shortest possible way. It is rather dry, sometimes ambiguous (not clear examples), and uses terminology that most people that are new to BI/DW will not feel comfortable with.
Chapter 1 is a good reminder of the complexity of BI/DW and there are many valid points (very briefly mentioned) but it offers little value in itself. I don't mind this chapter to be this way as this book is not an introduction to BI or DW, but I think author should say it and offer references as this is a point where "new to BI" people might just give up or skip it.
This chapter also contains the author's point of view on operational systems and suggests that if operational systems are constructed with BI in mind and include MDM (master data management) and track history then BI tools could simply run against operational systems. To a degree I agree but I don't think that is realistic. All data will never exists in operational systems (external data), we could ask all business users to use operational systems but this requires time (more than creating a spreadsheet) and would only be achievable in mature organization. I see this approach working for self-service BI but not for everything as we still need a dimensional model... yes we could do that with less effort but we need it as there will always be something extra that requires ETL or integration with other sources of data.
In the past, I also thoughts the same and I have started building my internal operational system with BI in mind and what happened? I gave up! I was spending way to much time on something that was bringing way to little value, and at that point I become "true" business user who makes a conscious decision to reject "proper" IT Development (due to constraints and little value). Bear mind I am very technical person so my change of thinking was natural and needed but not easy (took many years!).
Chapter 2 - Why Traditional Methodologies Don't Work
Mainly waterfall is discussed with plenty of good points. What the book doesn't mention is that author of waterfall in his document said that waterfall is NOT suitable for complex IT Projects so waterfall is greatly misunderstood by people and it took decades before companies started to realize or learn that it is NOT suitable for complex IT/BI Project like the author of waterfall said himself half a century ago! Good example of slow progress isn't it?
I presume waterfall was used because people didn't have alternatives and it seems in the old days people liked formal processes, definitions etc and these days anyone can pick up a book and start doing BI... I'm no exception... I don't have a degree as let's face it most Universities are way behind and I attended 3 Universityies and gave up 2 fairly quickly which supposed to be "top" universities (how naive I was these days) and one was very good but they didn't get funding to continue which is a real shame as it was real life learning where I did equivalent of 1 year and there is University of Dundee in UK that does BI and seems to have very good reputation and I have spoken with Mark Whiteborn and got really good impression but unfortunately again it is about ROI and with my experience, skills and alternative projects I can do; it no longer is something I want to do; mainly as I prefer to spend the time on other more valuable for me projects.
Coming back to the book ... Author shares an exercise she likes to do with people who manage projects which makes them quickly realize that they can't get everything! and I will try to use this exercise in my next role if I get a chance.
Today I hope to read Part 2 - Going Agile which has only 3 chapters so it probably will be just an overview and I hope to start Part 3 Extreme Scoping Planning Process which is majority of the book that hopefully will give me all answers I'm after.
This chapter focuses on Spiral Data methodologies which I agree with and makes a very important point that Vendors can build Data Marts in 30 days but cover only small % of true effort and many people (not all) will quickly regret the decision to make it quickly (low quality) but that doesn't mean there is no value in quick data marts and I believe there are needed as there are essentially a proof of concepts with comprises.
After reading Chapter 3 I have a feeling that the author mainly uses Inmon methodology and Kimball way is rarely mentioned and if is mentioned than comments are very misleading. There were a few occasions when Kimball would be a perfect fit but author doesn't refer to Kimball and Data Vault is not mentioned and I doubt it will. Spiral Data Methodologies sounds very good although I have some doubts and personally I would split it into two distinct phases so I would go definitely Kimball Bus Matrix (one Business Process at a time) and use it as a good representation of final outcome, do PoC (with PowerPivot) within several days (draft + ongoing) and let business people play with it and do high level estimates on integration effort and that point I believe a decision should be may to either do this particular Business Process or do a different one.
I believe this is important as people must make a decision which Business Process is most valuable for them and as the other underlines new approach WILL slow down projects in order to avoid low quality solution with high maintenance costs but I believe having alternatives for users is important even if there are short term... I know author says this should be avoided but I believe trying to do it properly first time is going to put too many constraints. Imagine a situation where you tell business people that from now on we don't build any new stand alone solutions and than the day business gets new source that must be integrated to make decision but are told to wait 9 months? This is simply unrealistic you have to have both long term and short term strategies for data if you only focus on long term than people will do short term anyway... why? Because they have to do their job with the skills and technologies they have available.
Let's see if my concerns are going to be answers in next chapters.
This chapter gives brief introduction to agile development and shows how it deals with key waterfall cons. Overall it was "positive" but carefully chosen sentences make us doubt this approach, which makes it even more strong for people who have experience with Agile (Scrum) on BI Projects. As I mentioned before Scrum is very good but not ideal which is why I combine it with Kanban and TOGAF but let's see what is the author alternative to Scrum for BI projects.
This was the best chapter so far! Previous 4 chapters were constructed in a very specific way to support chapter 5 and the contrast is excellent. So my slight annoyance changed back to true excitement :)
This chapter clearly presents reasons when Scrum/XP works but also it underlines why it is a problem (choosing easy path that most will regret in 1-2 years time) and shows when Scrum / XP doesn't work, obviously this all subject to "variance".
There were some diagrams and one shows Extreme Scoping however it is exactly the same as Inmon DW 2.0. and missed Data Vault and I have a feeling Data Vault will not be mentioned which would largely waste opportunity to make improvements to the methodology.
Spiral Data Warehouse was merged with Agile (=Extreme Scoping?) and I'm glad it happened as it just didn't feel right and updated diagram is more what I currently use.
New concepts were mentioned but I need more info to fully grasp it and it finishes with Extreme Scoping guidance where first point is always trust your instinct which I do! but I will rephrase it to use appropriate best practices but if it doesn't feel right or doesn't work than find something better (like Extreme Scoping?) or change it until it does make sense and works!
Last point is 14 which is have fun with your projects life's too short! and I agree but I would change it to: Do what you enjoy most! If BI is not for you or you don't like your role, company or people you work with than change your job! I've been doing it all the time and I have experienced myself which roles and environments don't work for me. I enjoy what I do a lot!!! but if someone puts too much constraints on me (force me to do it the wrong way.... e.g. Waterfall) than I don't enjoy but I change it by finding more suitable role :)
It's been a long chapter and in this chapter that gives a lot of practical advice and steps you need to consider and how you should structure it.There are plenty of good points but at the same time this chapter clearly shows Inmon way and has no consideration for Data Vault and Kimball which is a real shame as Data Vault can significantly simplify the processes and Kimball Bus Matrix is a great Data Road Map.
Main problem is ETL complexities and trying to do entire flow of data in one go where us I believe with Data Vault and careful consideration of reporting you can break it down and do independently and that means that if you hit a roadblock you can switch to something else without wasting your valuable resources but that requires a slightly different approach than is describe in the book.
Extreme Programming will work for organizations with serious budget but this is the same approach I noticed when reading Inmon DW 2.0. which was missing implementation details so Extreme Programming feels like add-in to Inmon DW 2.0. but would be very difficult to use for smaller projects. For example one of my projects was to build a data warehouse for a young company (<1 year when I started) using one Consultant/Developer (myself) from 12 different sources within 3 months? Obviously that is completely unachievable and Extreme Programming wouldn't work at all and the company didn't get a data warehouse but ODS with Snapshots and Kimball Prototype + Training. Was that wasted money? Definitely not as this new approach helped the company to solve key data issues (one place + option to use SQL/Cube) and was far better than anything else they tried and they learnt new things. Many companies have serious budget constraints and if you tell them the most accurate figure of how much it would cost to do Extreme Programming DW than it would probably produce very low ROI.
What BI is missing is approach that has both long term strategy and short term strategy. If you estimate effort and say it will take 2 years and we will deliver it in 8 releases and than someone ask what about my requirements they are important for me! Well not important enough wait 2 years! That is completely insane and people will switch to stand alone BI Yes the author is right they will and you know you shouldn't be stopping them! but you need long-term and short-term strategy which must co-exists, yes there will be waste but if I waste 50k to do stand alone app for this person who wanted it and this person non-important projects becomes a great success and gives 500k extra income per year than wasn't not worth it? Extra income can be used to push some of it to long-term solution with some re-work, yes re-work but if you plan carefully it should be minimized.
BI is complex but you cannot put these unrealistic constraints as it is like saying we will forget about the rest of business (who cannot afford it) on focus on this. You still have to control the rest which is part of Enterprise Architecture.
Coming back to the chapter a lot of details were provided but without recommendations so metadata is important and you can build or buy it but apart from that the book doesn't say anything else and I believe it would help if the author provided some recommendation like which software worked for her. Again a lot of what I read was discussed in Inmon DW 2.0. but on the other hand this focuses on delivery so it contains a lot of relevant points and information so overall it is worth knowing this even if I don't agree with everything.
Last comment I would like to make is there many documents which don't focus on value and quality and best documents are those that can be used by both Technical people and business people and I believe this approach can greatly reduced number of documents and improve the communication (clarity) and cut the time needed to implement. The book shows the standard (old) list and I believe with Data Vault and Bus Matrix this step can be significantly improved to provide better value and quality.
This chapter gave me new motivation to continue our Project KEBI.
This is one of the shortest chapters and as I mentioned before it is still too complex and tries to do the release in one go where us I believe it can be broken down using Data Vault and Kimball approach which focuses on delivering one business process at a time. Personally I believe Bus Matrix is key document between Business and BI people and can have great value if people understand it and I believe it is worth the effort.
I talk a lot about Data Vault but don't get me wrong I do not use it that often as the team needs to have skills to do it and support it but I believe this fairly new approach will eventually be used together with Kimball and Inmon so it is just a matter of years before that happens.
This was a very useful chapter as it detailed all activities and dependencies for each category and brief explanation of each activity. There is a lot of it but that is the most useful part of it as people often greatly underestimate it the scope of the project or miss certain critical steps. The second part of the chapter shows a practical example and a scenario is presented and then certain tasks are eliminated to get the final list of activities for the release. This approach is very good and it ensure that nothing is missed. Some of the activities are similar to TOGAF however these are more specific to BI projects so it is a very useful reference.
This was the most valuable chapter for me. It shows how teams should we organized and how they should communicate and it is different to Scrum which is a good thing as I never really like this aspect of Scrum, it's fine for small team (=one team) but doesn't really work between teams especially for BI projects and Extreme Scoping is the way to go but at the same time authors says that using Scrum/XP for "Development Tracks" is fine and I have to agree that that is a good combination.
It is suggested teams should be small and I agree the smaller the team the more effective they are. The worst mistake companies do is do big teams (uncontrolled growth) without any overall structure and plan.
In my career there was one aspect that I particularly didn't like and that was that a team would gather and discuss the issue and try to work out the problem and this is one of the most ineffective ways I've seen. The mix of people, different level of skills, experience, attitude and personalities without any structure for the meeting often results is one / two people wanting to do it their own way and the rest either joins the battle or just gives up (let them sort it between themselves). Extreme Scoping solves this problem by having core/SWAT team and let "Developer tracks" do it their own way (with compliance to standards, policies etc).
The chapter also gives a list of all possible roles which is very useful for reference.
I have enjoyed this chapter as it has showed me solutions for common situations that I knew could be largely improved and having a methodology that I can refer to that specializes in BI/DW Projects is going to make my life a lot easier!
This is a nice chapter that clearly shows who should be involved in each stage, responsibilities and which role/person is 'tie breaker'. It also shows how teams should be organized. One suggestion I really like is the one that no one works alone and that teams often have business users (joined responsibility for delivery) and I agree that this does help to solve the 'us vs them' (Business vs IT) problem which is a very common issues (attitude, expectations etc).
I like the brainstorming idea but at the same time I'm of opinion that it should be limited to small groups and subject experts (I strongly suggest to avoid brainstorming in large groups with mix of people at different competence level (exceptions apply)).
What hasn't been discussed in the chapter is quality of people you employ. This has been excluded as it is very delicate matter but generally I would think twice about internal promotions for technical roles for two reasons: most likely there will not be exact skills match and internal people may not be aware of best practices and common pitfalls. I mention that as I often see database (OLTP) people doing BI work and these are two different worlds so if someone will stick to their old role habbits than you may end up with something that is more an operational system and not a BI system. That is not to say you should never do internal promotions...You should! Just be prepared to do proper training which may take a while before in "sinks in" and learning on the job might be actually quite expensive in long term as it least consider one subject expert (internal or external) who can provide guidance and training.
This is full of great tips and shows how BI plan should be managed on micro and macro level. I agree with author that "Product Owner" (Scrum) does not work for Enterprise Level (not stand alone) and often is a problem when you have "competing" business units over next release slot.
I really like the "backward" approach and I believe many companies can avoid surprises if they use this technique. Different people track progress (informal) different and I like the author is fine in using Scrum/XP for non-core teams where Scrum/XP rules cannot conflict with Extreme Scoping. My only additional way to do it is Kanban as BI "tracks" have well defined phases and personally I use (with Katie) LeanKit.com as it is very simple but at the same time very effective.
At the end of the chapter author shares more openly her experience and gives extra attention to people (treat them well!) and admit that first release is never easy.
There is a comment about documentation and suggestion to auto-generate it after work which is rather common but personally I believe this kind of documentation has little to no value, as often no one will actually use it. My view to documentation is it is important but must be of very high quality and value, others don't do it at all!
Good chapter that discusses what changes companies need to take from organization perspective and it also discuss the uneasy aspect of cultural change.
It mainly is discussed around TDWI BI Maturity Model which interestingly is how I actually thought about BI (age) and summarizes key challenges and BI evolution (from BI Maturity perspective).
Overall I found the book to be very good that shares practical tips and wealth of experience but at the same time I feel like it didn't answer one critical aspect which is how to handle short-term needs of those who cannot wait 1-2 years before they request is taken care of and another aspect that I believe can be improved is the dependencies so instead of trying to do one release with everything inside relate to it maybe it is a better idea to split it but I believe this is only possible if another methodology is used for back end (Data Vault) and used together with Prototyping/PoC before and after Data Vault implementation. Additionally I believe Reporting should be independent release and more focus should be on self-service BI at early stage of implementation so reporting can start after core development is finished (not at the same time which is very difficult to do).
Before I finish let me answer my initial questions:
- Is Extreme Scoping really better than Scrum (which isn't perfect for BI / DW)? and how does it achieve it?
- Yes! but Scrum can still be used in dev tracks and it achieves in a very similar way as TOGAF. I never understood why people can think they can use Scrum only hence I used Scrum + TOGAF and althought Extreme Scoping is like Scrum (with adjustments) and TOGAF it focuses on BI/DW Projects and contains and it deals with key BI challenges so it much cleaner and easier to apply and I believe many companies will soon switch to it and I hope to be one of people who help this methodology grow their audience (especially in UK) as it really deserves the attention.
- What have I been missing or what have I not noticed or experienced yet?
- I share and agree on many subjects (except Data Vault and Short Term Solutions) but I have been missing some method like how to arrange teams (core + dev/extended tracks), manage projects (micro/macro) and a full list of roles and tasks which I found very valuable.
- Is Extreme Scoping going to add some value to TOGAF (which is an Enterprise Architecture Framework)? (I doubt it will try to make TOGAF redundant as it most likely will be very important part of BI/DW Projects).
- TOGAF is a framework of frameworks and Extreme Scoping perfectly fits in but at the same time you don't have to use TOGAF (I would still use it) as key elements are covered by Extreme Scoping.
- and most importantly how can Extreme Scoping help me with BI projects? and it is very convenient time for me as I will be starting a new contract at Ofsted on 12th of December 2013 as a Data Systems Architect (+ Trainer and maybe Team Lead?) so I hope to gain new knowledge and get better start from day one :)
- It's BI specific and has a lot of good tips and references so I'm sure I will be regularly checking the book for some tips.
I hope you enjoyed my rather comprehensive review which contained my personal point of view. If you would like to share your thoughts with me or start a discussion than feel free to contact me (social media account at the top).