I found myself in an embarrassing position the other day – I was pitching to a potentially important client, and he mentioned ‘Data Discovery Tools’. I had to admit that I had not heard of this term, despite the fact that I was pitching with a tool to help him understand his data better.
So I rushed to Wikipedia – the source of all knowledge. With some relief I discovered that the term was coined in 2011 as ‘end-user-driven approach to BI’. After quietly asking myself what other approach to business intelligence is possible, I investigated a couple of products in the field and discovered that they were attempts to allow users to use their spreadsheet techniques, principally pivot table methodology, to interrogate larger data sets than can be put into a spreadsheet, and use nifty graphics to show the results. I can see the value in that, even if it does not solve the problem that I’m trying to solve.
But that’s not the point. It is that it’s becoming increasingly difficult to keep up with the pace of development and the plethora of new products and methods in IT. By nature and inclination I am a generalist, believing that ‘the bigger picture’ will give me insight that I could not get by concentrating my efforts in one field. I’m also older, so my mind is cluttered with all sorts of information that is useless now. But I wonder it it is becoming more difficult now to create software that is genuinely ground-breaking, as the industry fragments into different areas of specialization, and new products tend to be improved versions of old products – mainly using the amazing power of modern computers to do jobs that would not have been possible even five years ago.
My interest, of course, is that I’m trying to do something much more radical. I believe that it is high time that systems used a different type of database other than the relational database. This is not to say that the relational database is obsolete (it will take many years for that to happen), but rather that it has proved to be ill suited to the business of data analysis, which is why there has become a wider and wider gap between IT using relational databases and analysts using spreadsheets.
I have yet to prove that my form of database really will become a better option for the analysts, and I’ll be writing many articles to say why that’s so, but here my question is the same as is asked of politicians – does a lifetime of specialization equip people to see the bigger picture? The quick answer is that, if the industry was entirely staffed by dilettantes it would be most likely that there would be duplication of developments, poor coding, and little real progress. However if, on the other hand, the only progress is made by those who’ve spent a lifetime in some highly specialized field, there will not be any real breakthroughs either. We need both types in our industry.