### It's helpful to have access to a lot of data, but what do you do if the data is too big?

![blue modern analysis computer Data Analysis Instagram Post.png](https://cdn.steemitimages.com/DQmdEWB67PR2w3bXv1gYkbDvkfPMDAWXYy7nwCfWaeDdkXG/blue%20modern%20analysis%20computer%20Data%20Analysis%20Instagram%20Post.png) *What do you think about big data? (Copyrighted to @tanzim2)* # Big Data: Think Big For years now, organizations have been toying with the idea of how to integrate data from a variety of sources in order to gain a better understanding of their customers, products, and services. And with the explosion of social media and mobility, the majority of data generated today is unstructured. [Source](https://www.communiquepr.com/pr-a-critical-success-factor-for-business-leaders/2460) As a result, "big data" has become one of those buzzwords that wraps up all unstructured data into neat little packages; however, many are now discovering that big data is much more than a buzzword. Big Data is the area of data analytics dealing with high volume, diverse and large-scale datasets. Big data was originally focused on computer data sets, but it has since expanded to include other sources such as healthcare records. It is an emerging field in which tools are still being developed and more organizations are utilizing it to manage their IT infrastructures. Big data is a term that refers to massive amounts of information that are evaluated for trends and patterns that provide relevant information. The study of people's online behaviour and interactions is given special attention. A study by Informatica and The Economist Intelligence Unit predicts we will have created and stored over 40 zettabytes of data. The study also estimates that by that time, 2.5 exabytes of data will come from video and 90 percent of all online transactions will be based on real-time analysis. Social media has been a particularly prolific source of big data for the past few years; however, other sources are increasingly being tapped for big data including sensor networks, smartphones, smart meters, digital HR systems and car dashboards. This data is then analyzed by systems that are designed to automatically detect patterns and trends in these datasets. ![what.png](https://cdn.steemitimages.com/DQmca7xpRMYbStWW8vfZYLVFgEe3s8ri2owEc5SM4Ef3xnH/what.png) *Confused about Big Data? (Copyrighted to @tanzim2)* The playlists and video recommendations provided by online streaming services such as Pandora and Netflix are examples of how big data is being used to better understand customer preferences. Basically, these services create music and video playlists that lead users towards similar choices. As the user interacts with available items, the system learns about the user's preferences based on those interactions. The system then shares that information with other users. It's a practice known as collaborative filtering. --- # The Larger Market From an intelligence perspective, this data can be used to find patterns and look for associations that may not have been previously known or considered. For example, in 2009, Netflix began creating lists of movies based on what was popular in a particular country. As the company gained more information on what type of movies appealed to its users, it began to notice a correlation between movie preferences and a country's borders. As a result, the company began recommending that people who streamed movies from a particular country watch movies from other countries. The issue with big data is that all this information needs to be processed and stored, which creates unique challenges. With the amount of data increasing exponentially every year, big data is forcing IT managers and analysts to rethink existing IT strategies. It has become much more complex to find ways to store and process all of this data in an efficient manner. Recent years have seen a shift to real-time analytics, where the data being analysed is close to real time. This transition has been made possible by better hardware that performs faster and cheaper memory storage. Development of faster hardware and software tools is increasingly being employed for in-memory processing of big data. Additionally, "small data" benchmarks can be used as a baseline against which to compare significant deviations from normal performance (i.e., abnormal). Small data also refers to datasets in which individual events are considered significant and where the entire dataset is small enough (comprising up to 10 GB) for conventional processing methods. ![Blue Data Analytics Instagram Post.png](https://cdn.steemitimages.com/DQmauyKou3kHYSeRN6JZdprDvpYdNhZkbCs2n5PESXjQwkM/Blue%20Data%20Analytics%20Instagram%20Post.png) *Big data is changing our lives (Copyrighted to @tanzim2)* Characterization of big data as "big" is dependent on context. Data that is considered "big" can range from datasets of a few dozen megabytes to petabytes or exabytes of data. Big data can be used to describe technologies and applications ranging from relational databases such as Oracle, SQL server and DB2, to various NoSQL solutions such as MongoDB, Apache Cassandra, Neo4j and the Hadoop framework. --- # Open-source Framework Integration Hadoop is an open-source software framework that provides the essential frameworks to run distributed applications on clusters of commodity computers. [Source](https://www.ibm.com/analytics/hadoop) It was created by Cloudera Inc. and Apache in 2004 and has been extended since then by a community of contributors, including Yahoo, Intel, and Nokia (formerly called Open Geospatial Foundation). It has many implementations across the computing industry that provide services such as search engines, web crawling engines, and data mining systems. It's amazing to see that as a data professional, you have to have the ability to conceptualize, demarcate and classify the different types of data and then use the appropriate tools for obtaining actionable results. In order to demonstrate how this works, here is a list of examples of both Big Data and Small Data. The difference between these two types of data is having large datasets or collections that are unstructured. [Source](https://www.quality-assurance-solutions.com/what-is-data-analysis.html) Small Data comes in many categories including mobile app analytics, location data on mobile devices, social network analysis, sensor network analysis, and other forms that can be applied to derive insight from small datasets. --- # Analytics Tools Big data analytics tools have been used for a number of industries including retail, healthcare, finance, and government. [Source](https://medium.com/@google_92930/ironorbit-managed-server-hosting-a901718379b4) In his book Big Data: A Revolution That Will Transform How We Live, Work, and Think, author Andrew McAfee describes those who are working in the big data world as "data scientists", a person who has the ability to design algorithms that make sense of large sets of data. This designation encompasses everything from simple manual processes using spreadsheets to complex applications that run on Hadoop clusters. The term big data also includes web analytics tools such as Google Analytics (GA) and Adobe Parature plug-ins for Google Analytics. ![Big Data.png](https://cdn.steemitimages.com/DQmQTqFg11ZTSR2JymuW1ujZud329jLfQX9Lx2kEohzAyYB/Big%20Data.png) *The Big Data! (Copyrighted to @tanzim2)* In addition to the tools used to organize and analyse big data, there are also other solutions being developed to help manage big data. One popular example of these solutions is Hadoop, an open-source software that allows data processing on large clusters with distributed storage and high bandwidth. Data is partitioned into different sets while also being indexed in a MapReduce framework. The result of this processing is a distributed file system that allows for fast access to raw data sets through traditional databases without implementing the additional infrastructure required for Hadoop clusters. Another recent solution for managing big data is Hadoop Performance Database (HP). HP, according to the company's website, is "an OLTP-style, high concurrency and high-performance database built on top of Hadoop." --- # Conclusion Big data is changing our lives bit by bit without realising. All the citizens of a country are getting connected to the web, and all this information can be used for several purposes. Currently big data is being used for detecting frauds and online crimes. There are a lot of other usages that also need to be explored and big data will surely help in improving the quality of life everywhere.

--- --- ## References: 1. https://datafloq.com/read/21-ways-will-big-data-improve-life/ 2. https://datafloq.com/read/5-ways-big-data-affects-your-personal-life/ 3. https://blog.bismart.com/en/big-data-is-everywhere 4. https://onlinedegrees.mtu.edu/news/ways-big-data-affects-your-personal-life