Monday, August 27, 2012

So That's What IT Recruiters Are Good For!

The BI Market seems to be fairly hot right now, at least in Australia, if the number of calls I'm getting from headhunters and IT Recruiters is any sort of indicator. However an experience I had this week makes me wonder.

Right now I have a need to ramp up the number of BI folks in our company to handle some project demands. It's a short term gig for the time being so we directly approached a few service firms as well as posting in one or two LinkedIn groups. Two days later I received an email from an IT Recruiter offering me the chance to interview for a great BI position with a first class company there were working very closely with. The position description felt very familiar and no wonder, I'd written most of it only a few days earlier. It had been augmented a bit to give the impression that what was in reality a short term engagement with a slim chance of extension was instead just the initial piece of a long engagement. A hunch had me checking Australia's main jobs website. Sure enough, two more recruiters were also advertising jobs which were also obviously the roles I was trying to fill. Both had extra or erroneous information, I suspect to make their version of the job more attractive than the one being advertised by their competitors.

Recruitment "consultants" taking initiative and creating opportunities to make income for themselves and their companies is one thing, but it causes problems for those of us working in, or managing those working in, the BI arena. With multiple parties now advertising my two roles now seemed like eight, so when those two are filled it will seem as if many more roles have been quickly filled. I suspect this contributes to the general feeling that demand is high and might even be artificially forcing rates up. Worse still will it cause a scenario where this perception of great demand draws people from other IT areas to re-train in the BI field, potentially causing a future glut of supply?

So let's be generous and say that recruitment consultants generate a buzz for us and drive potential candidates our way, but those of us with our own Human Resource Departments could do that anyway. What else do recruiters bring to the table? In my experience the majority of those I've dealt with lack any depth of understanding - matching skills and expertise to roles is haphazard at best, and even basic screening of applicants seems limited to "if they can spell BI then they're qualified to get a second interview".

So, for me I struggle to see the value..... But then I had an experience that made me realize that recruitment consultants do serve a valuable purpose - they're here to make us laugh. I was approached by a recruiter interested in my data warehousing expertise and keen to learn if I had a forklift license and what my typing speed was. I suspect data warehousing may have been a new term for him! Another, during a preliminary chat when I was trying to find a candidate to fill a new data warehousing role, asked me if I felt rising real estate costs would make warehouse space more expensive and cause my firm to lift our rates.

So, guys, keep the laughs coming, but you'll excuse me if I pass on your other offerings!

Friday, August 24, 2012

The Next Data Rock Star

I miss the 1980s, there was lots to like about that time: great music, a lifestyle less cluttered by technology, crazy hair, the fact that I could still grow enough hair to have crazy hair. For most of my adulthood if you had asked me which time in my life I'd like to go back to I'd have picked the 1980s. But now I'm not so sure, I'm starting to pine for the late 1990s through to the mid 2000s. Why? Because, that was a time when quite a few data folks working in the business intelligence arena were the rock stars of IT.

I'm lucky enough to have worked on my fair share of interesting and high profile data warehousing and BI gigs during that period. The problems I had the privilege of working on were often difficult, few (if any) commodity solutions or services had manifested, and the solutions we provided were often anything but cheap. However in solving those problems we added real value to the business, at times doing things that their traditional IT staff or software vendors had said couldn't be done or had tried (and failed) to solve with more application centric approaches. Perhaps most importantly during that period few BI projects were undertaken unless the business could really see the benefit. Maybe this was because of the high cost, maybe not. Whatever the reason, those of us lucky enough to be leading the charge and playing key roles in these projects often got the accolades of not just our IT and data colleagues but also  of key business stakeholders as well. I was fortunate to parlay this into a successful consulting career  with word of mouth securing a series of back to back BI engagements stretching for over a decade.

But I fear that the BI Rock Star is a thing of the past. It's becoming a crowded field. Any number of people now "do BI" and what was once bordering on black magic is commodity thanks to advances in (BI) software products and hardware processing power. Even worse, the phrase "reporting" is commonly be used almost interchangeably with "business intelligence". We've all too often moved away using BI undertakings to support complex decisions and solve wicked problems to one where often all too many reports are produced for banal reasons or reasons unknown. It's no wonder then that business stakeholders seem to be seeing less value from their business intelligence dollar.

So, if BI is indeed on the wane, I wonder where the next Data Rock Star will come from. If I was to encourage my kids into the field where would I point them? Will the Data Scientists be the next wave to be lauded by the business, or will those jumping on the Big Data bandwagon get that privilege? Maybe those playing in the NoSQL space are about to be thrust to glory, or perhaps the social media tsunami will mean that's it the intersection of all three which will be the sweet spot. I actually think it won't be any of these. Both Big Data and Data Scientists to me still feel like buzz words - hot topics - that the industry has thrown up and bandied about to the point where airline magazine syndrome has kicked in. I don't see the clear path to the wide stream killer need that means business stakeholders will clamour for solutions in these areas enmasse any time soon.

So, if you can see more clearly than I, I'd love to know what the next big data thing will be. I miss being a rock star!


Saturday, August 18, 2012

Data Quality - Running in the Badlands

I like to run a little. Actually, I like to run a lot. Back in the days when my work had me travelling most weeks one of the upsides was the opportunity to regularly find new running routes in new cities. Most times this resulted in an enjoyable run and the chance for a bit of sightseeing. However on occasions things didn't go quite so well. Several times I've felt very uncomfortable and one on occasion I had genuine fears for my safety.

Three such times stand out for their similarities although they occurred in different cities around the world - Las Vegas, Milan and Philadelphia. On each occasion I was speaking at a conference and so had a little time to kill, but not enough time so that I could run too far from my hotel. All of these cities promote themselves as either tourist or conference destinations and to the casual observer seem to fit this brief well - safe, busy, functional and (in two of the three cases) prosperous. At first glance it would seem that nothing is amiss and that the "service" they offer is well provided; they are "fit for purpose", if you like. However in each city less than six blocks from my hotel I found myself running through either seriously rundown areas, areas with (in one case) chain mesh fences and burning cars or areas where I was harassed and threatened.

Now, I admit that probably 99% or more of tourists or other visitors to these cities will probably never see the areas I did nor find themselves in a threatening situation. It strikes me that there are parallels with data quality in many business systems: the places where everyone goes, those areas that support the most common transactions and queries have had most obvious data quality issues sorted out. In most cases the people using these systems largely have no problems which stem from data quality issues so over time the perception develops that data quality is not an issue. Even when the odd data quality problem does crop up often its dismissed as an outlier or the pain of it forgotten before it can be used as a driver for a data quality improvement initiative. Much like the cities I visited all seems to be working smoothly as it should.

However on my problem runs I definitely didn't get the result I was after - I didn't enjoy sightseeing and the training effect I had planned for didn't eventuate as I either had to cut my run short or ran (away) at a faster pace than I had planned. Without doubt there was at least a loss of effectiveness and probably a failure to get to the desired outcome. The same occurs in our systems when data quality issues lurk just out of plain sight. When users venture into those areas, be it with ad-hoc requests or with infrequently run analytics, the outcome they seek may well either be missed entirely or hampered by inefficiencies. These issues have knock on effects too - perhaps someone reading this blog won't run next time they attend a conference in one of those cities or may even choose another conference entirely, in much the same way users may choose not to again wade through an analytic process in a certain area of data, preferring instead to rely on gut feel or their own sets of numbers from desktop spreadsheets.

So should we as data quality or data governance practitioners act to rectify these less mainstream data quality issues. The answer is probably "it depends". For me, it's a cost benefit question, if the impact of the problem is bigger than the cost to fix it then there could be a strong business case to be built for a fix. The bigger issue though is not knowing that the problems exist in the first case, so perhaps it is beholden on us to attempt to understand where these problems lie so we, and our stakeholders, can best choose where to focus data quality improvement efforts. Interestingly this very style effort actually is in play at the hotel I stayed at in Philadelphia. Seeing me walk back into the lobby in my running gear the hotel concierge asked how my run was and then offered me a pre-prepared map showing run routes and a very clearly marked border indicating the outer limits of where it was safe to run. I only wish they had advertised the existence of this data steward before I'd gone out running!