BI - It's Not Just Reporting

Many times across my career I've been frustrated by what is, in my opinion, a gross under utilisation of business intelligence tools, platforms and even functions and teams such as Business Intelligence Competency Centres. All too often significant investment has been made in purchasing software, building data warehouses and other solutions, training staff and hiring external consultants, only to have this considerable horsepower and potential being set to work on (just) "the reporting".

Building BI solutions often comes at considerable cost in both time and money and if the chief item provided is purely a set of static reports that tell us what has happened in our organisations then someone, somewhere (and probably somewhere higher up in the organisational chart) will sooner or later start to question the wisdom of the investment. And that's not just a problem for the here and now of the current project, but for the future too. If he has been tainted as to the general value of business intelligence then he may well push back against future initiatives in the area, potentially robbing the organisation of future competitive advantage or effectiveness gains. The problem is made worse still when the default approach to setting project scope is to examine the set of existing reports available from in and around the incumbent solution. In this case the company will have a shiny new toy and (maybe) a prettier set of reports, but may not have gained much in the way of additional business value or actionable insight.

We need to remember that BI offers the potential for much more than just reporting. This was true well over a decade ago and is even more true today with the technology and tool advances we've seen between then and now. Just look at the name: BI, that's Business intelligence folks - intelligence for the business. Simply knowing what happened (via a report), for the sake of compliance, audit or because the procedures say we need to have a piece of paper punched and filed on a shelf, really does little for the running of the business, let alone its advancement. BI should be about helping make decisions that better the business, that help it carry out its operations in the best way possible.

The term "decision support" may seem a bit old school, but perhaps it does capture what BI can offer to the business other than just "the reports". Recently I heard the phrase "the three tenses of BI" and it resonated with me. The first phase is historical - this is the looking back, telling us what happened, function of BI - these are the reports that so often get developed as the majority, or even the entirety, of a BI project. The second tense is the present tense. BI instruments in this area tell us what is happening right now. Reports can serve this function too, but often they miss a key component that stops them playing an effective role. Whatever is used here, be it reports, OLAP components or some type of complex event processing, the information conveyed must be provided in a form and in a time frame that allows it to be used by the recipients in a manner and a timeliness which causes a change which benefits the business that would not otherwise have occurred. The third tense of BI looks to the future - using not only internal information but also looking at wider information sets to try and predict what may happen in the future in and around the organization, thereby providing food for thought for those setting the strategic direction for the organization. What's important to note there is that the second and third tenses of BI require information and intelligence to be presented in order not just to enable decisions to be made, but perhaps also go so far as to highlight that decisions should, or even must, be made.

Simply having a great set of tools available isn't enough. If we want to move on from being perceived as just report writers then it's up to us, as BI professionals, to educate others as to the potential of BI, challenging and assisting them to find ways in which it can be better used across our organisations. So go on, take the first step - share this blog post :)

When I Grow Up I Want to Be a Data Scientist

Some time in the early 1970s I formed my first career aspiration. When I grew up I wanted to be a bus. Yes, you read that right - not a bus driver, but a bus. It seems ridiculous now, some 40-odd years later, but I certainly wanted it bad back then! The reason it didn't seem crazy all those years ago is that I really had no idea what being a bus involved and what's more a bus seemed like a pretty cool thing to be - lots of the books that I had read to me featured buses zooming all over the countryside having all manner of exciting adventures, there was at least one cartoon where one of the main characters was a bus, and buses even had songs sung about them. Why wouldn't a three year old boy want to be a bus?

Fast forward 40 years and focus on the data industry: big data is being talked about in server rooms, board rooms and everywhere in between, and the term Data Scientist is becoming more and more common. It's a position I'd never heard of twelve months ago and now it seems hardly a day goes by where I don't see it mentioned in a blog post, industry email, or white paper, etc. A quick look at a number of job posting sites across the web confirms that organisations are indeed looking to hire people into roles with the Data Scientist title. But does anyone really know what the job entails, is there an agreed job description, even in a general sense? Judging by the diverse descriptions of the job ads I looked at it certainly doesn't seem so. I've talked before about my concerns that the Big Data concept is very much an overloaded and poorly understood term and I worry that the new "must have" role of Data Scientist is facing the same problems.

But, so what? Does this situation even matter? I think so. I applaud anything that raises the profile of those of us that work in the data and information space. I still remember what it was like to ride the wave in the early days of business intelligence and data warehousing and if others can have similar experiences and success today due to the next big thing being data centric then that's a good thing. However, I worry if the  shining beacon of the Data Scientist role starts to lure people from other facets of IT, or potentially worse still, attracts students entering University whilst the speciality is still in its infancy. With the role a Data Scientist will play, and how they will deliver value, yet to become understood in a consistent way then how can we expect our higher education institutions to prepare courses of study that will equip students with the skills that they will need when they reach the work force? I wonder if we will see such courses morphing, emerging and disappearing over coming years as industry and the universities try to better find what is expected of Data Scientists and how to build the knowledge and skills base they need to be successful. What of these early students, the guinea pigs or crash test dummies of the coming wave of graduate data scientists? Are we leading them astray, promising them areas of work, challenges and career paths which no-one yet knows will actually be there in the same form (or at all) five or ten years from now? Let's not promise too much too early. Better people come to the profession with their eyes open and with realistic views of what to expect, than simply because of hype.

Enough doom and gloom. If anyone can really tell me what a Data Scientist does, I'd love to know. I actually might want to be one when I grow up.

Data Ownership and Data Responsibility Should Go Hand in Hand

Chances are that all of us who have tried to advance a Data Governance initiative will have had the data ownership discussion and socialized the idea that IT shouldn't own the data. In the earlier years of my data governance efforts getting acceptance of this concept was most often the first stumbling block. Common remarks from both the business and IT fronts reflected long established culture and practices in which data was seen as an IT focus - they managed it, wrangled it, invested and got to the root cause of data problems, and they held the responsibility of fixing data problems. This resistance, although frustrating, at least had a common theme which allowed me (and I suspect others like me) to build up a toolkit of arguments, presentations, stories and case studies to help move stakeholders toward an understanding that data is best owned by the business and not the IT function.

I've written about the need to keep working away toward implementing data governance before in my post on Thinking Like a Vogon. Having a bag of tricks built up and finessed over time is, in my opinion, key to successfully importing data governance at an organization. It allows those of us charged with championing data governance to keep working toward advancing stakeholder understanding and buy-in over weeks, months and years. Recently I've noticed a trend which has the potential to poke a hole in this bag of tricks, reducing its effectiveness and necessitating a rethink of how I engage with stakeholders. These days I encounter less push back when I propose the idea that IT doesn't own the data. The concept that the business owns the data now often seems to be easily accepted, with some business users going as far as saying "of course we own the data!"

Acceptance! It's all easy from here, right? Maybe not. While there seems to be a lot more widespread agreement that business users own the data this doesn't always translate into the best outcomes for data governance and improved data quality. Statements from stakeholders along the lines of "we own the data, but IT let us down because they can't get the data quality right and they don't understand our data" ring alarm bells for me. That one step forward may just have been followed by two quick steps back. What's missing here is responsibility. Ownership without responsibility is hollow and reflects an understanding which still has quite some way to go to reach maturity. I liken this sort of statement to a father boasting about the achievements of his children before reaching for the phone to arrange a nanny to look after them whilst they are home from boarding school. Just like we can't expect school teachers and others to take sole responsibility for shaping our childrens' lives and helping them through their problem times, business users can not claim to be data owners and then take a passive role only getting involved in data issues long enough to point the finger of blame at IT.

So how can we this new phenomenon be dealt with? Sure, a part of what needs to be done is updating the message and tailoring the tools and communication methods we use to address this new position. But, it's more than that, as data governance professionals we need to look for ways of showing those in the business with this view "what's in it for them". We should look to find success stories - show where there was demonstrable benefit gained when another business user started to take an active interest in the health and well being of their data. Even better, we should spend time with those business users who have already embraced not just data ownership but data responsibility as well, helping to make them into champions for the cause and assisting them to sell the benefits to their colleagues. The broader IT function also has a role to play here. We need to ensure that IT doesn't reinforce the existing culture, and as such there needs to be a little gentle push back when the business shows signs of pushing all data responsibility to IT. This need not mean a flat out "no", nor an abdication of any and all responsibility, but rather taking these opportunities to engage with business users, working with them as enablers rather just than in isolation as technical troubleshooters blindly trying to solve problems often without enough context or knowledge. By working together - making changes on both sides - IT will boost data context familiarity and those on the business side will get involved more often and earlier. From a management perspective, look to change IT policies if they talk about IT owning and managing the data, and think about establishing data reference groups.

No matter what approaches we take, there is one very important thing that can not be forgotten. We must help business people in their new roles as data stewards. We can not leave them to feel their way along. Build documentation and tools to help and guide them, recognise new responsibilities by modifying Position Descriptions and the way these people are measured and rewarded for performance. Not only will this signal that the organisation is serious about data governance and data quality, but it will also guide and encourage the stewards to perform better.

Why IT Can't See 2 Plus 2 Doesn't Equal 5 (And Why It's Not Their Fault)

How do a loaf of bread and a small bottle of orange juice lead to a blog post about data? Here's how. Last week I bought (or at least tried to buy) a $4 loaf of bread and a $2 orange juice. The young lady (a trainee, according to her badge) behind the counter cheerfully informed me that it would be $27.60 and seemed a little taken aback when I asked if she was sure about the price. She assured me it was right and even went so far as to turn her monitor around to show me - "See, there it is!". Risking the chance of becoming the customer from hell, I pushed my claim that the price was wrong. Another staff member (herself probably not old enough to have left school) was called to assist and tried her best to educate the problem customer - "Look, the computer says that's what it is, so that's what it costs". It took me leaving the goods on the counter and turning to walk out to have the duty manager come over and straight away see the problem and charge me the correct price.

I bet we've all had at least one similar experience and laughed about it, putting it down to "kids today" or asking "what are the schools teaching now?". But step back for a second and think - should we really be so quick to judge? Take a less critical view and ask what is really happening. The shop assistants are most likely casual employees, which means they spend only a few hours each week in that environment, they don't live and breathe the goods they sell and couldn't tell you the price of a loaf of bread without referring to a price list. While they can do their job and function largely without mistakes if enough boundaries are set and systems provided, when something goes wrong or when the pressure is on they lack the familiarity and deeper understanding which lets them sense when something (e.g. my price) is wrong. Chances are that the math skills of the two casual shop assistants were just fine, it was the lack of context of the individual prices and not yet having a confidence gained from situational familiarity which prevented them from recognizing a problem and speaking out with a different answer than that given by the system.

Is there a parallel between this example and how we work with data in our organizations? I think so! I've certainly heard (on more than one occasion) people from the business bemoaning that IT just don't get it, they don't understand the data. If the IT folks don't work with that data much, only a few hours per week, or in blocks of time many months or years apart, then that's not surprising. Even if they do work on the data more frequently they don't have the context gained from working with the data in the same way the business users do. What's apparent (at least to me) is that this gap in understanding is not the fault of the IT staff. If people don't work with a dataset closely and regularly enough to gain familiarity and understanding then even what are [to others] glaringly obvious mistakes may well go unnoticed. In the same way as the shop manager needs to find a way to better equip her staff to deal with situations like mine, we (as Business and IT Managers) need to find ways to better deal with data centric issues in our organizations. This may happen in a number of ways, for example

• embedding IT staff in the business to have them work with the data and alongside the business users 
• having business people take an active and accountable role in data migration projects and not leaving IT staff to make snap judgement calls about the data at the pointy end of a project to hit a go-live date, or 
• bringing the idea of data governance into the organisation and getting the business people involved with decisions around the data.

No matter which of these approaches you adopt I think two things should always be present. Firstly don't blame the folks at the coal face, and secondly introduce and promote the idea that data ownership belongs with the business not the IT department. Miss these opportunities and chances are that, just like me with my bread and OJ,  at some stage your company will be asked to pay a much higher price than seems reasonable.

Data Migration: Don't Ask What Should I Take, Ask What Can I Leave Behind?

I've written before about the importance of Selling the Data Migration Strategy and getting senior management buy-in. Recently I've been engaged in that sales job within my own organisation and just the other day met some resistance. On past projects disagreements around my strategic intent often took the form of objecting to my position that migration is an activity that should be front and centre for the business with them, rather than the IT or IS function, having accountability for its success. However, that wasn't the case this time, instead the contentious issue was that the strategy I'd written didn't explicitly call out the we should only migrate the minimum amount of data.

From my point of view this isn't something that belongs in an Enterprise Data Migration Strategy. It's not that I don't think that keeping the volume of data to be migrated down has merit. I don't need convincing that involving more data increases the cost and heightens the risks of overruns or impacts outside of the organisation's walls but, even more importantly, I don't think it's my place to be mandating a minimum data approach in a blanket fashion. For me, each of our data migrations will have different needs, different profiles and different levels of data to be migrated. And let's not forget that it's the business, not the technical staff, who should be making the calls about how much data to migrate!

This discussion got me thinking. There is value in keeping migration data volumes low so how could I, through the strategy, encourage this behaviour without presupposing it was the right course of action for each migration project? After some thinking I've landed on updating the data migration strategy to include a new principle:

Migrate the appropriate level of data. 

Expanding this a little, I see two key points that should be considered in finding this appropriate level of data.

• Firstly, a default position of migrating all available data is not where we should start. 
• Secondly, and following on from this, don't just ask the question "how much data do we need to migrate", but instead try the question "how much data can we afford not to migrate - what data can we leave behind?" 

With this question as the starting point I think there is a way of ticking all the boxes: the appropriate amount of data is migrated, the data migration volume is kept as low as possible, and IT doesn't drive how much data is migrated - the business still gets to make this decision.

The unanswered question here is how does the business decide on the appropriate level of data? I've used various techniques to help in the past and I've some new ideas as well, but that's a discussion I'll leave for another blog post.

Big Data: Are We Really Ready for the Final Frontier?

I've just read an excellent post from Rich Murdane entitled "Big Data" and the Failure to Communicate. I'm glad Rich penned this piece as I was beginning to think I was either alone with my position and thoughts or, worse still, just missing the point! Rich has managed to nicely put into words a feeling that has been gradually building with me. For a few months now I've had a growing sense of disquiet that "big data" was becoming a buzz word - a phrase that was on everybody's lips but often with little substance behind it. That is, folks are talking about it because they feel they should be talking about it, but in reality they (and we as the broader industry) have not yet taken the time (or perhaps more kindly, had the time) to really think through what big data is and more importantly what it means to each of us and what  value it might deliver to our organisations.

I fear there is a real chance that much money will be spent too early as folks jump on the big data bandwagon before they they really have a good handle on which part of big data they should be focusing on. If we look at what "Big Data" is then a number of things spring to mind. For me the the obvious one is condition monitoring data - generally structured time series data. It's not just the size of this data that makes it big, it's how fast it arrives - what's beginning to be more commonly talked about as the velocity of data.

For me, and the big data problems I'm likely to face in my industry, I think this may well be the key issue at least in the short term. Operational technology is prevalent and in some cases this could mean thousands of data points arriving per second - huge waves of data washing in. One of the questions that I think needs to be considered is how high velocity big data should be used, or perhaps how the best value can be derived. Quite a bit of the discussion I'm hearing and reading suggests that this question is not being asked, but rather there's a leap of faith being taken resulting in a default position that this data needs to be stored and analyzed. I agree that over time this might well be the case, but for many right now I wonder if this value proposition is understood well enough to be effective. Perhaps it might be wiser to wait and look at other ways to gain value from this style of big data, one of which might be a form of complex event processing to monitor the stream of data and pluck out those events or short term trends which allow an action to be taken to materially impact operations. Those of us without a compelling business case to store, and later interpret across, huge data volumes might be best to wait and watch, taking action not because we feel that we should be, but because we understand what the value is and how we will unlock it.

This watch, monitor and wait brief should be wide enough to make sure that all forms of big data that might deliver value are considered. This means that it's not just the structured data area that is important, unstructured data is perhaps even bigger. For many of us with long pedigrees in the data space this may be somewhat of a new and unfamiliar area. I think we have no choice but to step outside of our comfort zones and become more familiar with this space. Much of the short to medium term value to be derived from big data, in my opinion, will come from this area for many industries. Sentiment analysis and mining of data from social networks is a great example of this which is already becoming important for many organizations. It's also an example where a stream of data can be monitored for data if interest as well as gaining value from analysis across larger pieces of historical data.

Big data is here to stay. It's not a matter of if it will impact your company, but now only a matter of when. You'll have to jump in and ride the wave at some time, just be sure you don't jump into too soon. Make sure you haven't worn out your funding and senior management support, will and trust before there is a real opportunity for you to harvest value from big data. Big Data may or may not be the final frontier. I could borrow from Star Trek, but I'd rather borrow from 1980s one hit wonder Star Trekkin'. It's Data Jim, but not as we know it. Good luck in your explorations, but be careful. There may not be Klingons on your starboard bow, but there is almost certain to be an executive or two asking the hard questions if his money disappears into a black hole with no value return.

Data Migration: Pushing Risk to the Vendor is not the Answer

Data migration exercises are risky. I've previously blogged about steps that can be taken to reduce risk in my posts on Selling the Data Migration Strategy and Data Migration: Track Record or Process Maturity. One area that I've not touched on in either of these posts, but one that has come up as an idea in pretty much every data migration project I've ever been involved with, is the use of the commercial negotiations process to handle data migration risks. Here I'm talking about payment milestones being closely tied to successful migration of the data. This must be a good thing, right? If our vendor doesn't get our data across to the new system then we don't pay them, or we withhold a sizable chunk of cash. I'm not so sure - for me this approach is at best a blunt instrument, something to beat a vendor over the head with, to cajole them into getting the job done.  

Step back and analyse the position you'd be in if you exercised the right to withhold some part of payment and you may start to share my point of view. OK, so your data migration project has hit problems, it's either running over time or the vendor has flat out just failed to deliver. That's not good, but at least you've got a good chunk of the project budget left, you've not wasted your company's money, or have you? Take a look at the projected benefits for the data migration project and / or the larger project the migration is part of and consider which of those benefits has either been forgone or deferred to some future date due to the migration problems. How do the financial outcomes look now? Sure, you may still have $200k in the project budget courtesy of the non delivery clause you negotiated, but you're going to have to spend that (and then some) to attempt the migration again, or fix what you have, with another vendor. Also some or all of the (financial savings) benefits the company was counting on in this year's budget now won't come through until next year (or later). Not the best situation to be in!

Of course, the counter argument can be made that by having financial penalties built in to the contract, or payment milestones tied to delivery milestones it's more likely that the vendor will be motivated to ensure that the job gets done, that your data gets migrated no matter what it takes. It's those five words which bother me - "no matter what it takes". What this may well mean is that vendor is sacrificing the true desired outcomes of your project in order to protect their financial interests. There will always be a degree of this apparent in any vendor- client relationship, but this is much easier to manage when there aren't extra pressures looming in the shape of potential financial forfeits. Shortcuts, a focus on short term fixes to specific problems rather than techniques in line with the best overall outcomes, and circumventing process and testing in order to hit deadlines are all behaviours you may need to deal with if you insist on these sorts of clauses. 

But what about the scenario where the vendor is the party suggesting this type of arrangement? In the past I've seen vendors offer arrangements where the lion's share of their payment (as high as 90% of the project total) is not due until the migration project is complete and accepted. You could argue that in such cases there is no harm in using these terms, after the vendor is offering the option freely, rather than being forced to this arrangement. In reality, the vendor may be being forced to these terms, not by you (their client) but rather by market forces. They may have to offer such arrangements in order to win work. Often this will be the case when they are inexperienced / new to the market space or have failed to build a portfolio of past successful projects which they can use as reference sites. There will be exceptions, but often the vendors who need to offer these types of terms are perhaps the vendors you least want to have on your data migration project. 



So, what's the alternative? In my opinion it's to be proactive. Don't push the risk to the vendor, but step up and own that risk yourself. Recognise and publicise the data migration risks early and take steps to mitigate them. It's your data so take accountability for the success of the data migration process. You can't afford to be simply a passive by-stander through the migration. If you are, when things fail (and they probably will at least to to some degree) yes, you'll be able to point the finger of blame at the vendor and say they failed to deliver.  But you'll still have a failed project on your resume, your company will have missed some or all of its projected benefits, and you may well have an extended miserable time as you attempt to mop up behind the project and live with work-arounds for years to come.  A data migration strategy, upfront planning, carefully selecting a vendor who is aligned to your strategy and likely to be able to deliver because they work effectively to an established and proven process, and staying actively involved throughout the migration are far more likely to keep  your risks from becoming realised issues. 


So, be brave, step up and own that risk. Good luck!

Implementing Data Governance: Think Like A Vogon

Recently I received a note from a prior colleague. He was in the midst of trying to implement data governance at an organization but was getting severe pushback, basically he saw he was headed for project failure. I had a couple of pieces of advice for him which I'll share here as I'm certain he's not the only person struggling to get data governance off the ground.

Establishing data governance at an organization is not easy, it will take time and chances are that the first approach you try may not work. In my opinion if you're working on something with a name along the lines of "Data Governance Project" then you've got a problem. For me, establishing data governance isn't a project. It's not something that can be up and running in a few short weeks or months, but rather I think of it as an initiative - something to be worked on over time, building toward an objective with the flexibility to change tack or even retreat, regroup and try again depending upon the appetite of the organization and the receptiveness of the executive and key stakeholders.

Which brings me to my first piece of advice: Think Like A Vogon. If you weren't lucky enough to be a teenager in the 1970s and 1980s then you may not know what a Vogon is so a little back story may be in order. The Vogons appeared in Douglas Adams' book / radio play / TV series The Hitchhikers' Guide to the Galaxy. Their battle cry was "resistance is useless!". If you're going to embark on a data governance initiative then you need to adopt this as your mantra. You need to keep telling yourself not to give up, you will strike resistance, but you can't stop at the first hurdle (or the second or third for that matter) . Keep telling yourself that resistance is useless because you will establish data governance in the end!

The Vogons used their battle cry as the answer to any dissenting argument or position. Their common approach was to simply keep shouting "RESISTANCE IS USELESS!" at whomever didn't agree with them.

This brings me to my second piece of advice: Think Like a Vogon, Don't Act Like a Vogon. Use the mantra as a means to keep you going when you face pushback or you struggle to get buy in for your ideas, not as something to beat people over the head with. If your only trick is the compliance line, e.g. telling / forcing people to comply with data governance rules because they are the rules then chances are you've missed the value pitch for data governance and you're going about it for all of the wrong reasons anyway. Stop telling people they have to comply and start telling them what's in it for them if they do come along for the ride. Show what past problems might have been avoided had some data governance been in place or find how some small form of data governance will drive some demonstrable short term savings and go and implement that small piece. Sell the value!

The Vogons were not just a race of bureaucrats, they also wrote poetry . Unfortunately this poetry was commonly thought to be the third worst in the universe. Essentially their poetry was  nonsense verse and decidedly unpleasant to be subjected to.

Which brings me to my third piece of advice: Make Your Artifacts More Appealing Than Vogon Poetry. If you have a ponderous Charter, a rambling Terms of Reference or information that the Data Stewards will find it hard to make sense of, then invest some time in making it more useful. Make your communications with your stakeholders clear and consistent, make your value pitch concise and tangible and construct the material for your stewards so that it's functional and easy to use.  My final piece of advice is to make the data governance initiative visible. Let people know about it and what it can do for them. Spend some time and effort raising the profile. If you can get others excited by its potential then you may get support, and even pull, for data governance from within the business. Don't be afraid to let others know about your data governance initiative. Believe me, it's not something to be hidden away in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard."

I'll leave you to your Vogon Googling. I'm off to find a Dentrassi to make me a nice cup of tea!

Data Migration: Track Record or Process Maturity?

What's more important when it comes to engaging a vendor in the data migration space - a track record of prior success or a demonstrable maturity of both company and process? I suspect the answer will differ from person to person and project to project. Having recently been involved in a few vendor selection exercises I've spent quite some time pondering this question.

The obvious response might be to favour the vendor with an enviable track record of successfully delivering their past projects. Surely putting your faith in an organization that has consistently and regularly delivered is a good thing - right? Maybe not! I'd advocate considering a few other things before signing that contract. For one, how long is that track record? A 100% success rate is impressive, but what does it really tell you if the company is a new player or is entering a new market space? It may be an indication of things to come, but it could equally well be a reflection of the firm's (and her people's) early stage enthusiasm and a "do whatever it takes" attitude that has got those projects delivered.   If this is the case, is this what you want for your project? A successful delivery may be the end result,  and maybe what people see when the look at the project I years to come, but what might it look like on the inside whilst the project is in motion? Throwing people and dollars at a problem to get a project over the line can bring undesirable side effects for those involved. Increased working hours, last minute ad hoc requests, a scattergun approach and a feeling that true outcomes may have been sacrificed to the Gods of the Gantt Chart all take their toll on the morale of those on the project - both on the vendor and customer sides. People can and do burn out. Your best internal people may jump ship, recognizing what's coming, while others on your team may tough it out but carry the scars of their experiences from the project back into their normal roles, potentially causing distrust or reluctance toward adoption of the new system as a by product of their attitude. Factors like these can and do hit project costs but, perhaps more importantly, also can have a larger impact on realization of the projected benefits of the project. The unfortunate truth here, though, is that often success will be recognized and rewarded when a challenged project finally gets delivered, with no one noticing that the really important outcomes were missed. Be careful what you measure! But I’ll leave the discussion of selecting appropriate KPIs and their impact on driving behavior to another blog post.

What’s the alternative? I’d advocate looking for evidence of process, and not just for the existence of a few pages talking academically about the process in a tender response, but some indication that that process has actually been used in an organization’s prior data migration engagements. When this is apparent (and verifiable) it’s usually indicative of a level of maturity.  Chances are that in reaching that level of maturity the vendor has learned a thing or two. Among the ones that are likely to materially matter to you are how to ensure their staff make good use of the process rather than just blindly follow it, and (hopefully) the fact that the process has been born of experience, that the scars and successes of past projects have all had some bearing on what will happen on contemporary data migration projects (yours included). Having such a process will mean the success of your upcoming data migration project is less dependent on the knowledge or experience of one or two key people but instead allows a broader group to be able to make choices and take effective actions. Actions which are likely to be well aligned with the desired end result and not just knee jerk responses to a current problem. This is especially important as time and delivery pressures come to the fore as the go live date looms closer. Going down this path should also mean that there is far less need for heroic efforts to deliver the project. Those long hours that many of us recognize from past migration projects should be the exceptions rather than a daily norm. Your people will be less likely to burn out and / or become jaded with the new system and you’ll avoid the creep of toxic feelings that may otherwise have undermined the system adoption. And who knows, with all that good stuff going on you might even successfully migrate the data AND get good benefits realization!

Dumb, Dumber and Dumberer: Or how to make sure your data migration problems really hurt.

A little while ago I blogged / ranted about the importance of keeping the data migration activity close to the main project and not allowing it to drift away into its own little world. I used an example of a company that services me which had been through a recent data migration process and how choices it made with regard to communications with customers had apparently been made without regard for data migration issues, with the end result being data quality issues which might otherwise have gone unnoticed being made very obvious to the customer. If you like you can read the original piece here: Dumb and Dumber, or how to make sure your data migration problems get noticed.


Fast forward three weeks and I received my next quarterly bill from the company. No folks, that wasn't a typo - the bill for the next quarter arrived only three weeks after last quarter's bill. A quick glance at the bill confirmed that the data migration problems hadn't been resolved. That's OK, it had only been three weeks after all! But, amazingly, I got the same extra communications and marketing collateral I'd received last time that had inadvertently drawn attention to the data migration and data quality problems. Ok, I can understand how a lack of an holistic approach to the new billing system rollout could have resulted in this happening last time, but surely the flood of calls they must have got to their call centre after the last bill cycle must have alerted something that something was amiss. Do these people need a sledgehammer to drive the point home? Or, more likely, is it symptomatic of how bad the isolation of data migration activities are - a problem for IT to  own and solve. (Watch this space for a blog post on that issue as well!)

So, what's to be learned here? The key [new] takeaway for me is that data migration activities don't end at the conclusion of the go-live weekend. Having the data loaded into the new system is only one milestone on the journey. Someone needs to continually monitor the quality of data  and assess if it continues to be fit for purpose in its new home. This should include monitoring indicators from around the organisation that might provide clues that there are data issues afoot. And, in today's world where social media abounds, this monitoring really should extended outside the organisational boundaries as well. Chances are the marketing department already has some sort of monitoring of social media, so connect in with these folks and work out from there.

As a footnote (and perhaps not connected to the data migration) the company had also failed to reconcile my payment of the last bill against my account and so included a demand to pay this prior amount immediately. However they were nice enough to point out that if I was having trouble paying my bill they have options to help.  I'm contemplating if I should send them a reply along the lines of "Thanks for your offer. All good here, but are you experiencing difficultly migrating your data? Please contact us for a confidential discussion on how we may help you. Information architects have Data Migration Policies and can arrange suitable remediation plans for immature organisations."

But it's not all bad, at least they provided an example of how brands and reputations can be damaged as a result of poor data migration (and related) decisions. That's material for another blog post. You can read it here: Selling the Data Migration Strategy.

Selling the Data Migration Strategy.

There are two types of data specialists in this world: those who have been through a failed, or severely challenged, data migration project and those who will go through one in their future. Stick around in our field long enough and you'll join the former group, and it may not even be your fault. Even with the best intentions in the world, the foresight to see problems well ahead of time and a plan mapping out a path which has business users, data stewards and processes owners heavily involved and engaged you may still fail!

Why is that?

In reality there are several reasons but I'll drill in on just one here: the lack of senior management buy in and top cover for your data migration strategy. And I'm not talking about a data migration strategy just at the project level, but rather a data migration strategy set at the enterprise level. You do have one of those, don't you?

If you don't, do yourself a favour and start putting one together. Without such a strategy you'll likely find yourself fighting the same battles over and over,  having to justify your choices in the face of tight project budgets and deadlines which were set without regard for your tacit data migration principles, years of hard won experience or contemporary data migration best practices. You really can't [solely] blame the project manager who put the plan together either, after all what guiding documentation did he or she have to consult when they built the plan? Good PMs will look to be guided by existing strategy and principles documents if they exist and excellent PMs will probably seek out those people within a company with migration experience and specialist skills to see what they can leverage. I'm lucky enough to be working with one such PM now and have worked with a select few others in my time - you guys know who you are! But unfortunately the vast majority won't take such steps, preferring instead to depend upon their own past experiences or tasking business analysts to come up with a number - a hard task without specialist skills, experience and guidance. I've worked with far more than a few of these in my time - you guys probably don't realise who you are!

So, you need an overarching data migration strategy document to influence and guide behaviour when it matters - both early in the project planning and design stages and also later on as the pressure mounts close to the go-live date.  But, that's not enough. You also need to sell it to senior management, you'll need their support when the pressure hits. That pressure will come, there's no doubt about it. It may strike at different stages  depending upon the type of organisation you're working with. If you're faced with the task of introducing the concepts that data migrations are business, not technology issues, that business leaders cannot be passive in the migration and that ownership cannot be divorced from accountability then chances are you'll strike push back from business managers who don't want to lose key people to the project, don't have budget for business as usual back-fill or feel that the approach is just IT trying to abdicate responsibility. Alternately you may be faced with a aggressive timeframes or budgets which need to be challenged to ensure that you give the migration at least some chance of success. And, at some time a go-live deadline will mean that pressure will  come for data migration processes to be sidelined in a frantic rush to just get the data in any way you can.

When these things happen you'll need someone with real authority in your corner. So, how do you sell your overarching data migration strategy to the higher levels of the org chart in such a way that you don't just get moral support, but true buy-in? Chances are it will vary from organisation to organisation and even from person to person, but here are three ideas:

• Link the failure of data migration to delayed or foregone project benefits realisation. There's plenty of research out there that can show this linkage and whilst you won't know any specific numbers until the next major project business case appears you can at least plant the seed. Bonus points if you can look back in your company's history and quantify the amount of unrealised benefits for a prior project due to data migration issues. 
• If your company has culture of, or current focus on, safety then draw out how failed migrations could feasibly lead to a major safety incident or death.
• Show how data migration issues lead to damage reputation and brand. There are plenty of cases where this has occurred. I can count two in my city in the past three months alone.

Ideally back up your case with real world examples where a lack of data migration strategy, or a strategy which differs from yours in key elements, contributed to major data migration failures and manifestation of one or more of the above points. You'll find these in the press, case studies and academic papers, but perhaps the best source of such information will come from Auditor General's reports into failed or problematic Government projects as these will be both detailed and publicly available. Also gather some evidence of successful projects which were guided by a strategy which is well aligned with the one you've developed. These can be harder to find. There are academic papers and industry case studies, or you can leverage your network to gather these stories (and hard evidence to support them). Of course, if your strategy is well researched then chances are you may already have amassed this material as you built the strategy. 

Your data migration strategy should be a living document. Take the opportunity to learn from each data migration project, consider these learnings and adjust the strategy as and when there is benefit in doing so. Also keep your eye on the wider area as well. You may only do two or three data migrations every five years inside your company, but there will be many more going on in the broader community. Take the opportunity to learn from those where you can. To that end, I'm always keen to hear what others are  finding has worked (or not worked) for them. Drop me a line with your pearls of wisdom or to share your data migration battle scars - I'm keen to evolve my strategy too!

Findable, trustable and believable. How FaceBook Slipped Up

If I had to sum up the core of what's important for me in the unstructured data area right now I'd use three words: findability, believability, and trustability. Given that my word processing software has just drawn squiggly red lines under each of them, I'm not even sure they are official words. But they should be - without these three things users won't be totally comfortable using whatever piece of software, system or web portal we give them to work with and will probably disengage with it, at least in some fashion.

Don't believe me? Even the big players miss the mark sometimes. Look at FaceBook as an example. I suspect that the recent rumblings and user backlash against the new interface wasn't just because the look and feel changed. The experience changed as well. People didn't know where to look to find things and even if certain things were still possible. For me I wasn't bothered by the change to the look and feel, but I was still thrown by the change. Little changes make the difference. I no longer saw links I posted from external
sites showing up in the my general news feed. Did that mean that the post had failed, or could other people see them and my own posts where just filtered out of what I saw, or had I done something wrong - selected the wrong privacy settings, perhaps? My level of trust had dropped, and dropped quickly. So much so that I took to using other means to share links in some areas, resorting to emailing links around or using specialist sites like Garmin Connect and Strava to publish my running and cycling information. Changes to FaceBook's iPhone interface also left me wondering where to look to find certain posts. I now had a taxonomy of sorts which I could use to refine and filter what displayed on screen. I could choose to view all stories, just see photos, status updates, links, pages, posts from close friends, posts from people in my area or post from people who were associated with the University I attended, etc. Refiners! Great, less noise to wade through, give me just what I want to see. That's a good thing right? I thought so too, but again I didn't always see what I thought I should see, leading to a lingering doubt. Again a trust issue - was I seeing all of the data available or did I have to look somewhere else or do something else to bring it up? Findability had suffered as well (or at least I perceived it had).

Translate this into the corporate world and this might mean that if users lose trust in how their unstructured data is being stored or made available to thrm that they'll stop using a document management system or content management system and resort to emailing documents around the organization and / or putting content on file shares or in other information silos. Even in a well used and hitherto trusted system small changes can quickly result in users working around the system, taking us back toward the unstructured data anarchy that many of us are slowly working to bring order to. Slip behind on that challenge and suddenly your information is fragmenting and findability suffers as well.

What lessons are there to be taken from this? Firstly, small changes matter. Something seemingly insignificant can send ripples out which can set your information management goals back months. Secondly, user interface changes can and will impact on information management and information integrity and quality. Don't make design choices without considering the impact on the wider issues and your information management goals. Choices here may not seem important but can at times be just as critical as deeper information architecture questions or taxonomy design. And finally, communication is key. Make it clear what changes mean to how information can be found and what information is and isn't available and perhaps problems like these, and the negative perceptions that come with them, can be avoided.

Have you lived through similar issues? My medium term goals for information inside our organisation rely heavily on getting findability, believability and trustability right so I'd love to hear about what you faced and how you tackled it.

Rethinking Your EIM Program

I've not looked at a Gartner Hype Cycle in a while, but I'm pretty sure if I did I'd find Enterprise Information Management (EIM) Programs heading into the Trough of Disillusionment. From what I'm hearing these programs are doing it tough and its not surprising.  EIM aims and value can be hard to articulate, I know I've struggled at times with ours . If you have an EIM program what's your elevator pitch? Phrases like improved decision making, better data management, and leveraging the data asset abound but chances are that many decision makers are coming to see those as little more than motherhood statements, particularly if there's been an EIM program in place for a while with few real benefits yet realised.  Although there may have been initial agreement around the problems to be solved and excitement around the promise of the program your sponsors may no longer really be signed up for the journey and are probably now struggling to see the way there.  


Perhaps it's time to rethink the strategy we take toward reaching our EIM goals. Rather than one big program maybe the use of strategic themes might work better. Use themes to chunk your nebulous program down to smaller quick projects which deliver toward and enable a common theme and focus the efforts and imaginations of people around a few key areas and projects at a time. It's easier for people to see value along the way yet (if you keep your eye on the longer tern prize and larger goals) still possible to get to the bigger picture by drawing synergies from projects along the way. Some of the themes I'm promoting right now include making information findable and believable, and the use of templates to tie information and process together and allow nimble reuse when new business initiatives arise. 


Using strategic themes also means it's much easier for people to "get it" and you, as the architect of the broader EIM initiative, will have better control over how they perceive what's important to do next. Equally importantly you'll have a better chance that they won't "get the wrong end of the stick" and become focused on the wrong things and put their energy into work that really delivers little to advance your true EIM aims.  Another plus is that it's more likely that you'll have people wanting to get involved (and hence get better people) if they can see what is going to be delivered in the smaller projects within the theme. 


If your team structure allows it then look to embed account managers in to the various business areas to get early signals and information about what needs are out there and weave these into the strategic theme sets where appropriate. Doing so will get people on the train.  Let the inertia build and then use that inertia to drive through those pieces of work which are less tangible - whether they be advancing data stewardship, building out the enterprise data model, or something else.


I believe this chunking and strategic themes approach will become more important across the coming 12 -24 months,  particularly if your company places more stock in initiatives with strong business cases and has an over representation of business improvement projects among the successfully funded initiatives and pushes those with less obvious or quantifiable benefits to the back of the queue. This approach will also give you the flexibility to be nimble and stay aligned with business strategies and priorities as they change rather than locking you in to a full year's budget cycle many months before the first work even begins. 


With this approach there are bound to be trade offs but, hey, at least you're moving forward and making (some) progress toward your EIM goals. And that's got to be better than the program being killed entirely. 

Where’s the BI Value Proposition?

I can remember a time when the pitch for BI could (almost) be made along the lines of reducing headcount – “put in this reporting solution and you’ll no longer need the team of people who currently spend all of their time compiling numbers in spreadsheets for others in the business to use”. Even when BI cost a mint this scenario still offered an organization the chance to come out in front financially. These days things aren’t so simple. Despite decreasing costs for BI the sales pitch is probably harder today than it was a decade ago. And I’m not just talking about vendors and consulting companies selling BI products and services into their prospects. I’m also talking about selling the value of BI inside companies – making the case for a BI initiative to internal funding boards is getting tougher and tougher with shrinking budgets and competition from other projects and programs.

To have the best chance of success the value proposition needs to be clear.  Often now BI projects will be seen as providing productivity support. They won’t save headcount but they may free up some percentage of time for resources throughout the company or help those within the organization make faster and / or better decisions. Proposing cost savings based upon these time savings is a dangerous endeavor. Such savings may not be viewed as real given fixed wage and salary costs or, perhaps worse still, the department may simply have next year’s budget cut by that amount to ensure the proposed benefit is locked in for realization.

Coming up with the appropriate value proposition may need you to stand back and honestly examine what’s within the scope of the proposed BI program. Is it just static reporting or are you largely replacing existing reports like for like due to a burning platform? If so then perhaps it’s time to swallow your pride and pitch the program like an infrastructure project. Chances are your client or your internal funding board looks at the criteria for assessing these types of projects differently than others, recognizing that they are unlikely to deliver strategic value, cost savings or drive top line growth, but rather just have to be done to sustain the business as usual state – to keep the lights on, if you will.

If your proposed initiative is broader and can be aligned to more tangible business objectives then you have a number of things you should be mindful of when building the BI value proposition:

·      Value is a subjective thing. Look at the perceived value of the initiative at different levels of the organization. Do they all see the value, or does your push to realize enterprise value actually make things harder for one or more departments? If you can find win-win scenarios or at least avoid win-lose scenarios then it will be easier to get agreement on the value.

·      Can you show how the value can be measured over time (and is it actually going to be possible to measure this value)? Financials are just one aspect of this. Speed of decision making, greater visibility and transparency and reduced risk might be things to consider. What ever you decide to include make sure that you can measure and articulate the current state. Not only will it give you a benchmark to measure against but you’ll also have concrete examples of the issues you are trying to solve, rather than just pitching the value of the initiative based around academic or theoretical concerns or problems that have manifested in other industries.

·      Is all of the value expected to be realized in a short to medium timeframe? Chances are this isn’t the case, especially if the project involves analytics or data mining. There likely to be a perceived drop in value whilst the design and implement stages are underway and the business resources are having to participate in workshops, field questions and test while holding down their day jobs. The perceived value is likely to spike once the initiative goes live as the low hanging fruit are harvested but chances are the perceived value will again dip as the business gets more complex and costly due to changed and changing processes, new ways of working and information overload. The real value pay-off may well not come until this stage passes and this state of anxiety and uncertainty can be resolved by working with the business to work with these new learnings and new information to further enhance the analytics and any new or changed processes to again simplify the way people work. Expect this, plan for it and build it into your value proposition. If this pushes the benefits realization out too far to the right then look for ways of chunking the program down into smaller tactical projects all building toward the strategic end game. Show the early value and paint the alignment with the strategic aims and the later value that will be realized.

And of course if you’re coming from the IT side of the fence don’t forget to work with the business through all of this. Not only are you likely to have covered off more stakeholders prior to the pitch, but chances are you’ll reap a higher value from the finished product as well.

Oh, and if you’re from the business side, bring the IT guys in on things too!! Involving them early will lead to higher longer term value.

Dumb and Dumber. Or how to make sure your data migration problems get noticed.

Recently I received a bill from a company which went to great lengths to make sure I knew they had just replaced their billing system. Not only did their bill now look different (with fonts and colours that probably helped pay this year's private school fees for the children of some design studio director) but they also took the time to include their quarterly magazine (complete with an article on the new system) and a double sided full page insert explaining the layout of the new bill.

All good so far. Help the customers through the change, and all that! Glancing through the insert I noticed that loudly called out was a section which showed the amounts of prior bills and trends - last bill, this time last year, etc, and how this could be used to help customers monitor and control costs. Cool! Being a data guy I went straight to this section on my bill..... only to find nothing but zeros! Discussions with other customers in passing over the next few days told me that several lacked any historical data, some of them had historical data, although often for services that they didn't own and some lucky souls even had current period data for services that had long ago been terminated.

Now those of us that have been through a data migration or two know that they won't be without challenges and that mistakes will occur and some data issues may manifest. Calling these out is a good thing. Customers and users affected by the problem will likely be a lot more accommodating if they know about a problem in advance and are kept in the loop about progress toward rectification. But that's not what has happened here. Rather a feature which could have been a selling point may well have turned into a negative as it alerted customers to potential problems with their bills. At best, even if the data issues were nothing more than missing historical data a potential good news story was wasted and I bet that the company's call centre spent a lot of time fielding enquiries from concerned customers.

So how did this one slip through? Were the data problems a surprise to the company, a black swan that crashed into them after the system go-live, in which case the testing process has failed dismally. Or, were the data problems known but the system put into production anyway due to timelines which couldn't, or weren't allowed to, slip to the right? If this latter scenario was the case why was the extra promotional material sent out too? Surely, a short statement acknowledging the problems and talking about the anticipated fix times would have been more appropriate. Better still why not hold off on the release of the new paper bill layout until the issues had been sorted? This way no-one would have spotted a problem and few people would have been materially impacted.

What lessons can we take from this? The obvious ones we've seen before: Data migrations take time and they will strike unexpected problems - allow both time and money to deal with this when they occur. Plan data migration testing early, expect it to take numerous iterations and allow time for these. If there is not time during the go-live activity for comprehensive data migration testing then be sure to run a full dress rehearsal and test its outputs thoroughly. Perhaps not so obvious, is a very important lesson. Do not allow the data migration to become divorced from the rest of the project. Consider the impacts of data migration events and, when and if needed, take steps in the wider project to ensure that migration issues don't compound and lead to embarrassment, brand damage or unnecessary costs.

Single Source of Truth, Nirvana and a Good Run Spoilt

This post came out of a presentation I recently gave to a group of colleagues. I’d promised to put something down on paper to draw out the key points for that group so thought I’d take the opportunity to share with the wider community at the same time. 

When presented with the question “is a single source truth a good thing?” how would you answer? For most people, particularly those of us with business intelligence backgrounds, the intuitive answer is likely to be “yes, of course”.   We’d answer in quick-fire fashion because it’s just one of those things that we know to be true. In a similar fashion we’d quickly voice our agreement with the statement “there has been almost no good music released since 1989”.  Wouldn’t we?

Unexpectedly, I recently had my morning run violently collide with a single source of truth issue. Before I left for the run I loaded up my iPod shuffle with a playlist drawn from the Classic Rock genre within my iTunes library. Half an hour into my run all was going well, the sun was shining and I was happily singing (puffing) along to the sounds of The Police, Queen, Peter Gabriel, Iggy Pop and Nirvana. Hang on, Nirvana? In what universe is Nirvana classified as Classic Rock? So thrown was I by this heinous data quality issue that I had to physically stop running to jump to the next track. Talk about poor data quality having a huge impact on productivity!!

But what could have gone wrong? Surely iTunes is the best single source of truth for information about the music from its own library.  But the problem is just that – it’s a single source of truth but its expected to serve a huge diverse audience. How can we expect this one source and one version of the truth to suit us all? While I’d put Nirvana into the Grunge genre I’m sure many others would agree with the iTunes view and call it Classic Rock, while some of the current crop of teenagers would probably relegate it to the Golden Oldies genre.

For me, one potential answer to this problem is to draw different information from different sources, taking what’s the best fit for a given use and perspective from whatever the most appropriate source is and bringing it together into a composite view – a 360 degree view of an item. A key point to note here is that some of these sources of truth may not be the system of record, there may actually be more appropriate data hidden on a PC under a desk somewhere.  The critical data should, and perhaps, even must come from the system of record, but don’t rule out other data. In my iTunes example I’d expect items like the track names, the album name, the artist name and the year of release to all come from whatever central source feeds iTunes as these are all hard and fast and should be indisputable. But as for the rest, let them be drawn from elsewhere. Let people be able to take genres and ratings from sources which match their perspective and have it surfaced together on their screen with the core information. If one of those sources is unofficial then there’s an opportunity to bring it into the tent so that there are some controls and governance around it, and this should be explored.

Now, for all I know this iTunes data mash-up may already be possible with some combination of settings in iTunes and on my iPod, but I’m just a dumb user and I couldn’t find it in the 30 seconds that my patience allowed me to look for an answer. In a business scenario our users aren’t going to spend time looking for ways to work with the tools we give them either. We need to make sure what we give them is flexible, easy to use and intuitive or they’ll find other ways to work with the data, potentially opening the door to a raft of data quality and integrity issues in so doing.

And as for those of you asking why I had Nirvana in my iTunes library in the first place. There was a passing moment in Philadelphia in the 2003 when I was listening to them, but the attraction has long since passed.  We’ll leave the discussion of my failure to implement a decent Information Lifecycle Management Policy over my music collection for another day!

Oh, and by the way, if anyone does happen to know an alternate source of truth which presents genres as perceived by the average forty something middle class man please drop me a line J