Epignosis

The Enterprise Data Architecture, Part 1

With the explosive growth of the Internet and more recently, the Internet of Things (the rise of the machines…!!!), there has been a corresponding explosive growth in the amount of data available and collected from all those connected devices and software.  However, along with the vast volume of data that has been generated, there has been an even greater challenge in dealing with the data.  As early as 2001, various IT analysts started reporting on potential issues that would arise due to the mass amounts of data being generated.  Most notably, this article by Gartner analyst, Doug Laney, summarized the various issues attached to Big Data (as it’s commonly called) into whats known as the 3Vs:

• Data Volume – The depth/breadth of data available
• Data Velocity – The speed at which new data is created
• Data Variety – The increasing variety of formats of the data

Now, 16 years after that article was written, we still face many of the same challenges.  Even though technology has improved to be able to handle larger amounts of data processing and analytics, the amount of source data has continued growing at exponential rates.  Cisco releases each year a Visual Networking Index which shows previous years data as well as predictions regarding the next few years as well.  In their 2016 report (Cisco, 2016), the prediction was that by 2020, there would be over 8 billion handheld/personal mobile ready devices as well as over 3 billion M2M devices (Other connected devices such as GPS, Asset Tracking Systems, Device Sensors, Medical applications, etc.) in use, consuming a combined 30 million terabytes of mobile data traffic per month.  And that is just mobile data.

So what are the real challenges being faced due to this exponential growth of data?  Here are some facts to consider as posted by Waterford Technologies:

• According to estimates, the volume of business data worldwide, across all companies, doubles every 1.2 years.
• Poor data can cost businesses 20%–35% of their operating revenue.
• Bad data or poor data quality costs US businesses $600 billion annually.
• According to execs, the influx of data is putting a strain on IT infrastructure. 55 percent of respondents reporting a slowdown of IT systems and 47 percent citing data security problems, according to a global survey from Avanade.
• In that same survey, by a small but noticeable margin, executives at small companies (fewer than 1,000 employees) are nearly 10 percent more likely to view data as a strategic differentiator than their counterparts at large enterprises.
• Three-quarters of decision-makers (76 per cent) surveyed anticipate significant impacts in the domain of storage systems as a result of the “Big Data” phenomenon.
• A quarter of decision-makers surveyed predict that data volumes in their companies will rise by more than 60 per cent by the end of 2014, with the average of all respondents anticipating a growth of no less than 42 per cent.
• 40% projected growth in global data generated per year vs. 5% growth in global IT spending.

Datamation put together this list if Big Data Challenges of which I want to highlight a few a specifically.

1. Dealing with Data Growth – As already mentioned above, the amount of data is growing year over year.  So a solution that may work today, may not work well tomorrow.  Investigating and investing in the technologies that can grow together with the data is critical.
2. Generating Insights in a Timely Manner – Generating and collecting mass amounts of data is a specific challenge that needs to be faced.  But more importantly, what do we do with all that data?  If the data being generated is not being analysed and used to benefit the organization, then the effort is being wasted.  New tools to analyze data are being created and released annually and these need to be evaluated to see if there are organizational benefits to be gained.
3. Validating Data – Again, if your concern is generating and collecting mass amounts of data, then just as important as processing and analyzing the data, it is important to verify the integrity of the data.  This is especially important in the quickly expanding field of medical records and health data.
4. Securing Big Data – Additionally, the security of the data is a rapidly growing concern.  As seen in recent data hacks such as the Equifax hacking attacks, the sophistication of hacking and phishing is growing at a rate equivalent to the volume of big data itself.  If the sources of data are not able to provide adequate security measures, the integrity of the data can be questioned as well as many other issues.

Big Data is here to stay, and it’s only going to get bigger.  Is your company ready for it?

References:

Cisco (2016). Cisco Visual Networking Index. Retrieved from http://www.cisco.com/c/en/us/solutions/service-provider/visual-networking-index-vni/index.html