![\"Marcel](\"https:\/\/miro.medium.com\/fit\/c\/96\/96\/2*Aghq2jgtZepKIyOWy-MDVg.jpeg\")
<\/div>\n<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
AI, Machine Learning, IoT and Cloud-Based Services Must Deliver Value From Their Data.<\/p>\n With growing attention devoted to AI, machine learning, and IoT, what we\u2019ve come to know as big data has become an even broader version of itself. In recent years, big data was seen as an unstoppable force of nature that would either overwhelm enterprises or propel them to new heights.<\/p>\n This next generation of big data \u2014 we\u2019ll call it expansive data, pulsing through systems in real-time, powering processes unseen to human eyes, and adapting and learning as it goes along \u2014 is going to reshape enterprises in ways not even anticipated.<\/p>\n This requires attention to new types of tools, platforms, and approaches to deliver value to today\u2019s data-hungry businesses. Expansive data will represent ever-growing volumes of information, potentially increasing within enterprises at a rate of up to 36% a year, according to Dresner Advisory Services<\/a>.<\/p>\n This next generation of big data \u2014 we\u2019ll call it expansive data, pulsing through systems in real-time, powering processes unseen to human eyes, and adapting and learning as it goes along \u2014 is going to reshape enterprises in ways not even anticipated.<\/em><\/p>\n<\/blockquote>\n Platforms supporting this growth include Amazon Web Services S3<\/a>, Spark SQL<\/a>, Hive<\/a>, and Hadoop<\/a>. Additional tools popular in enterprises are Apache Spark and Tensorflow. Expansive data places even higher demands on enterprise infrastructures, processes, and the managers and administrators responsible for making it all work.<\/p>\n Organizations are Leaning Heavily on Data Assets<\/p>\n That\u2019s because organizations are leaning more heavily than ever before on their data assets and analytics capabilities, and initiatives such as AI and machine learning, to help them compete. Edge computing is also a defining factor in expansive data.<\/p>\n There is likely to be greater activity at the edges \u2014 expansive data means more processing may be distributed across IoT networks. Data can be ingested, processed, and even stored within edge devices and systems, and, if it is deemed critical on an enterprise-scale, moved to centralized data centers or clouds.<\/p>\n Edge computing continues to extend its capabilities. It encompasses a broad assortment of devices and systems that may require real-time interactions and responsiveness, including kiosks, autonomous cars and trucks, and sensors embedded across IoT. With comprehensive data surging across all points of the enterprise, infrastructures could be quickly overwhelmed with ingestion, processing, and storage demands.<\/p>\n Expansive data could also be valuable data without proper preparation. Fortunately, none of this is happening in a vacuum, and other developments may be helping organizations manage the challenge. Thanks to the ubiquity of cloud-based services, from infrastructure to platform to applications, the power, and capacity to support even bigger data environments are readily available. A new generation of database platforms and tools \u2014 led and enabled by machine-learning initiatives \u2014 is supporting the continuous, relentless data growth.<\/p>\n Hadoop is a big data framework that made huge-scale data analytics a reality for every company that will benefit from processing data. However, the software is beginning to show its age. While Hadoop was once seen as the single cure-all for significant data challenges ten years ago, today\u2019s expansive data calls for a variety of tools, platforms, and frameworks to help enterprises better manage their data. Nonetheless, the Hadoop Distributed File System can either support or be a part of data lake architectures, opening up a new mission for these environments.<\/p>\n According to a 2018 survey conducted by Unisphere Research<\/a>, a division of Information Today, Inc., 44% of enterprises had Hadoop in production, which represents a downward shift from 2016, in which 55% reported using the framework (\u201c2018 Next-Generation Data Deployment Strategies Report\u201d). Also, the survey found general satisfaction levels with Hadoop are mixed:<\/p>\n Only 14% consider themselves to be \u201cextremely satisfied\u201d with Hadoop, while 64% are either dissatisfied or lukewarm toward the framework. While Hadoop provided one-of-a-kind functionality in its early days \u2014 such as parallel processing and management of a variety of data types \u2014 other technologies and solutions also now share these capabilities without the skill levels that Hadoop demands.<\/p>\n Predictably, the growth of expansive data is likely to track closely to that of IoT itself. Accordingly, next-generation data technology initiatives represent new approaches to data management. The Unisphere Research survey found notable growth in the adoption of data lakes \u2014 places to store diverse datasets without requiring to build a model first. Their adoption continues to rise as data management personnel seek to develop ways to quickly capture and store their data from a myriad of sources and in various formats.<\/p>\nThe Expansive Data Generation<\/h2>\n
\n
Big Data and the IoT<\/h2>\n