Big Data: A Brief Historical Description

Big data is a voluminous dataset from a variety of sources and of varying types that is continuously generated at a high velocity. It can only be managed by specialized processes different from the traditional relational database processing tools in order to infer any relevance or insight from it and to establish the veracity of the data.

In other words, big data is a huge collection of data from different sources and of varying types that are growing at an exponential rate with time with no traditional means of storing or processing it but are valuable for insight.

The description of big data above stems from the historical use of the term up to the recent definition of parameters (i.e. V's) that are being used to capture the various aspects of this sort of dataset. My history consideration for my description starts from various uses of the term between 1997 and 1998 which are similar to the current usage to mean information explosion starting with Michael Cox and David Ellsworth in 1997, John Mashey in 1998, and Francis Diebold in 2000.

Also, the description considers the volume of the data, the variety of the data, and the velocity of the data which are credited to Doug Laney from Gartner's early attempt to describe the term big data. Other considerations in my description of big data are the value that can be inferred from the data as well as the veracity of the data. The value from the veracity of the data could provide the benefits of product development, predictive maintenance, fraud and compliance, operation efficiency and drive innovations in general.

Big data could be structured, semi-structured, or unstructured. Even though it is noted that above that big data requires non-traditional processing, the data could still be structured like traditional relational data. This data is referred to as structured data. This is data with a fixed format. The semi-structured big data has a structured form but it is not fixed or defined. The last type is unstructured big data which has an unknown form or definition.

In conclusion, examples of real-time big data are data from the Internet of Things like Alexa, home security cameras, social media site interaction, etc., for which big data is used to provide real-time recommendations based on current views, and news recommendation sites. Another example is weather prediction from continuous data from weather sensors and atmospheric information, health information from wearables, live transport information, and fuel management from autonomous vehicles.

Reference and Further Reading

• The Origins of big data
• What is Big Data?
• What is Big Data? Introduction, Types, Characteristics, Examples