Big Data

Posted: August 27th, 2021

Student’s Name

Instructor’s Name

Course

Date

Big Data

Introduction

Big data is presently regarded as one of the newest innovations in business technology. It has significant relevance in the fields of business, finance, research, and biological sciences. Advancement in mobile devices, communications, computing, digital sensors, and storage has necessitated the means of data collection. Therefore, large corporations’ existence has prompted an increase in total amounts of data across the world. Many organizations continue investing in big data with the realization of its importance in aiding the organization growth and performance. Specifically, the availability of explanations associated with the 3V of analyzing big data like volume, variety, and velocity brings about a means of defining data through varied state-of-the-art processing technologies. In this regard, this paper discusses Big Data, ranging from its architecture and infrastructure to the involved methods of data processing technologies.

Background Knowledge on Big Data

Big data is usuallyunderscored by three essential qualities: volume, variety, and velocity (Janev 121). Indeed, these three features exhibit substantial data volumes, varied information types, and information velocities. Regarding the data volume, Nielsen can generate approximately 300,000 columns of ongoing information at each successive second (Janev 121). On the other hand, more than one billion records of conducting extensive information investigation exist at a particular month. There are extensive investigations for both organized and unstructured information concerning the type variety, which assist organizations to generate experiences from diverse sources like customer exchanges, stock observation, store-based videos, sales management, and financial information (Janev 121). For data velocity, extensive data investigations are utilized to empower continued access and sharing of data to enhancesignificant data dynamics. Therefore, big data presumes its usefulness in diverse fields like science, research, building, medication, and remedial services.

State-of-the-Art Big Data Processing Technologies and Methods

With many organizations having failed to utilize the operational data, Big Data’s architecture must perform equally well to any company’s supporting infrastructure. The reason is that a large amount of growing data has emanated from varied unstructured public or private sources, such as machines or sensors (Hashem et al. 11). Precisely, this reason is attributed to the fact that most companies did not capture vast amounts of data in the past. Equally, one might blame the unstructured data sources on the unavailability of necessary processing tolls for generating outcomes within the stipulated timelines. In this regard, the implementation of Big Data technologies has contributed to improved performances in business modeling products and decision-making processes (Hashem et al. 11). While the following technologies and methods offer deeper insights into Big Data’s application, they have likewise facilitated reduced hardware and processing costs and further improved the timeline for checking the value of Big Data before investing resources.

Batched Based Processing Technologies

Apache Hadoop’s use offers an opportunity for the processing of large amounts of data like the case of SwiftKey, 343 industries, redBus, and Nokia. Hadoop is accustomed to performing intentional data applications because it utilizes a Map or Reduces programming model to process a large amount of data (Hashem et al. 13). Ideally, this kind of programming model operates vis-à-vis the divide and conquers system as a way of breaking down a problem into smaller units. The Hadoop infrastructure has master and worker nodes responsible for dividing and distributing small units of tasks.  Moreover, Skytree Server is another batched-based processing valuable technology in handling large amounts of data at high speed by offering a command-line interface (Hashem et al. 13). Skytree Server deals in real-time data analytics since it is operationalized to apply machine deep learning algorithms. Apart from that, Talend Open Studio is also a batched-based processing technology responsible for offering a graphical platform for extensive data analysis and applications (Hashem et al. 14). Thus, whereas it resolves big data queries without using Java language, its processing technology is relatively slow.

Technologies Based on Stream Processing

Stream processing depends on the choice of real-time technologies like Storm, Splunk, SQL Stream, and S4 (Hashem et al. 14). For example, Storm stream processes data by distributing real-time computational systems because it possesses both master and worker nodes. Although Storm cluster is easy to use with any programming language, its processing performance is less efficient and reliable than other stream processing technologies (Hashem et al. 14). Besides, Splunk stores indexes and further interconnects real-time data intending to produce alerts, reports, visualizations, and reports from the repository. Therefore, Splunk is presented in many ways, such as Log Files. However, S4 offers a pluggable platform for efficient utilization of the unbounded information streams (Hashem et al. 15; Sun et al. 11).Thus, S4 is considered a better stream processing technology for processing Big Data because it reduces latency through the use of local memory instead of the I/O model.

Big Data Processing Methods

Since it is costly to retrieve blocks from an extensive database index, it helps apply the Hashing technique. Hashing method might help recover data from a disk, even without the use of an index structure. Indeed, hashing performs well on discrete and random data since it is quick in reading(Hashem et al. 15). However, this technique is unfit for retrieving data prearranged in a particular manner. Moreover, the location of data from large and complex datasets requires the help of the Indexing technique since it applies varied sets of indexing on databases like semantic indexing, r-tree indexing, and bitmap indexing (Hashem et al. 15; Sun et al. 11). Equally, Parallel computing is a better method of processing Big Data because it uses various sources of data simultaneously. With its combination with Hadoop, it improves the processing power of sharing similar data among numerous servers (Hashem et al. 15; Sun et al. 11). Although this method entails fast processing of data, it is however occasioned with frequency scales.

Examples of Big Data and Case Studies

Big Data analytics continuallyplays a growing role in scientific research.While sensors have become cheaper and smaller, there has been a need for extensive scientific experiments that would improve data collection(Pence 161). More so, the collection of large data is further analyzed via Big Data techniques. For example, the development of the Square Kilometre Array (SKA) in Australia and South Africahaving more than 36 small antennas simulated single but massive radio telescope. The telescope is widespread over more than 3,000 km via numerous radio-frequency interlinked together, thus creating a powerful telescope array called interferometer (Pence 161; Sun et al. 11). Upon completion, it would be one of the largest and most sensitive interferometers in the globe. Consequently, the project is anticipated to gather one Exabyte of data in a single day, hence improvingBig Data analysis. The 17-mile diameter Large Hadron Collider (LHC) development at CERN in Switzerland is expected to impact Big Data analytics positively.  The LHC Data Centre processes approximately 1petabyte of data each day, bringing about fast storage, processing, and analysis of data (Pence 162). Most importantly, Large Hadron Collider holdsabout 150 million beams. These beams have power capability of 40 million deliveries in a second.As a result, more than 83,000 physical diskshave been successfully stored (Pence 162). Subsequently, 10 GB of data are transferred from the servers at a single second during peak rates. 

Advantages and Disadvantages of Big Data

Big data plays a critical role in improving the affairs of many businesses. People’s lives are enhanced by facilitating the way interactions are done (Sun et al. 10). Hastening processes usually work as a significant chance of increasing the significance of big data. Imperatively, the incorporation of the Internet of Things (IoT) is considered the newest involvements of big data. For example, sensors could generate essential data and information that assist in decision-making processes (Sun et al. 10). Also, big data has the advantage of decentralizing the decision-making processes, thus making use of it as a relevant opportunity for businesses. In this, business can utilize the data to expand its operations through informed decision making. The reason is that big data systems seem readily competent since they have been proved efficient in earmarking better decisions.Equally, big data provides an opportunity for the utilization of predictive analytics and visualization of data. Indeed, the process of undertaking decisions is attained via objective facts and figures (Sun et al. 11). Thus, the process of analyzing data analysis seems easy and straightforward. Consequently, big data is advantageous in allowing access to the most diverse data without difficulty.

On the contrary, big data is associated with some disadvantages because data is created at a very fast pace. Specifically, data utilization is essential and central for undertaking critical decisions (Sun et al. 11). In this regard, only 0.5% of the created data is collected for analysis and utilization, implying some cons of having big data (Sun et al. 11). As much information might be made applicable, some problems interconnect with big data utility in many fields. For example, if the initial data is flawed, the results will be false-leading no matter how efficiently they can be processed and analyzed (Sun et al. 11). Subsequently, garbage data would generate garbage outcomes. Thus, this assertion implies whenever wrong data is used in the process, the results will obviously be wrong.

Conclusion

With increased data processing rates among many organizations, Big Data technologies have received a considerable amount of thoughtfulness from IT communities. Whereas the batched-based processing technologies are efficient in collecting, storing, processing, and retrieving outcomes, they equally have limited resource utilization capabilities. Moreover, these technologies and methods of processing data focus centrally on the velocity of retrieving results in a short time. Besides, the development of the Square Kilometre Array (SKA) in Australia and South Africa is a Big Data case study with the objective of increasing processing of data up to Exabyte daily. In brief, as much as Big Data has the advantage of strengthening businesses’ performances through quick decision-making processes, it faces challenges associated with the utility of resources when processing and retrieving outcomes.

Works Cited

Hashem, Ibrahim A. T., et al. “Big Data: From Beginning to Future.” International Journal of Information Management, pp. 1-19.

Janev, Valentina et al. Knowledge Graphs, and Big Data Processing. State-of-the-Art Survey. Springer, 2020.

Pence, Harry E. “What is Big Data and Why is it Important?” Journal of Educational Technology Systems, vol. 43, no. 2, 2015, pp. 159-171.

Sun, Huidong et al. “Identifying Big Data’s Opportunities, Challenges, and Implications.” MDPI, pp. 1-20.