Big data encompasses the ever expanding structured, unstructured and semi-structured data that companies of all sizes collect and store. Most companies store their data in traditional relational databases such as Oracle, Microsoft SQL, and IBM DB2, Informix, PostgresGIS etc.
As the data grow in size companies start to find it very difficult to manage, analyze and handle Big Data. One main reason is because relational databases are not design to take advantage of horizontal scaling which is common in most big data technologies.
On the other hand, big data technologies require massively parallel software running on tens, hundreds, or even thousands of servers.
Big data technologies are still evolving and companies that have really made good use of big data technologies include Google, Facebook, Twitter and Yahoo!
Only a handful of companies in the oil and gas industries have implemented Big Data technology in their daily work flows. It is worth noting that companies do not need to invest in Big Data if there is no need to do so, since many traditional relational databases hat can handle large amount of data without any problems.
Big data technology bottlenecks include:
- New architecture;
- Experts or new skill sets;
- Data stack is complex and highly fragmented;
- Difficult to know where to begin; and
While big data technology is still evolving, current software licenses available can be divided into three main categories which include:
- Proprietary e.g. Oracle, IBM, Teradata;
- Open source e.g. NoSQL , HPCC, Hadoop; and
- Cloud services e.g. Amazon Elastic MapReduce and Google APP Engine.
Big Data technologies therefore describe a new generation of technologies and architectures, designed to extract values from very large volume of a wide variety of ever changing data.
Today, new technologies exist to realize the value of Big Data e.g. MPP, Hadoop, MapReduce. These technologies have emerged to solve some of the challenges posed by Big Data and provide new insights to business. These technologies are complimentary and the way most companies are leveraging the capabilities is by integrating operational databases with Hadoop.
By combining Big Data and advanced analytics in Exploration and Development activities, managers and experts can perform strategic and operational decision-making. Some big names in the industry have already implemented big data technologies e.g.
- Chevron’s proof-of-concept using Hadoop (IBM BigInsights) for seismic data processing;
- Shell’s pilot of Hadoop in Amazon Virtual Private Cloud (Amazon VPC) for seismic sensor data;
- PointCross Seismic Data Server and Drilling Data Server using Hadoop and NoSQL; and
- BP’s implementation of IBM Big Data model.
Advanced analytics can play an important role in improving productivity in unconventional, conventional and midstream operations in oil and gas. Big Data has come to the oil field not just for the big oil companies but for all mid- to small sized companies that have actually driven oil discovery in the US and at large. A Bain & Company report states that by implementing Big Data technology, oil producers can capture more detailed data in real time at lower costs and from previously inaccessible areas, to improve oilfield and plant performance.
The same report also states that taking advantage of advanced analytics could help oil and gas companies improve production by 6% to 8%. Small and mid-sized oil companies need to take advantage of some the benefits of big data analytics in upstream, midstream and downstream operations which could lead to the following:
- Enhance exploration efforts;
- Access new prospects;
- Improve drilling accuracy;
- Enhance oil recovery;
- Real time production optimization;
- Improve safety and prevent risk;
- Performance forecasting; and
- Cost optimization.
An example is BP’s Center for High- Performance Computing (CHPC), located in Houston, Texas. The BP Magazine reports that BP uses Big Data technology in its upstream sector,
“the CHPC facility helps the seismic imaging teams to simulate, process and predict what will happen in a reservoir. It does this by processing and managing huge volumes of geological data from across BP’s global portfolio, helping teams to see more clearly below the Earth’s surface. It helps reduce the amount of time needed to analyse large amounts of seismic data and can also enable more detailed in-house modelling of rock formations before drilling begins. With field developments costing billions, this knowledge is invaluable and its pursuit puts BP at the forefront of seismic advances.”
As reported by the Wall Street Journal, Royal Dutch Shell has implemented Big Data technology aboard its Noble Bully 1 in offshore Gulf of Mexico to help extract oil in once inaccessible regions of the deepest oceans. This new design has helped Shell drill wells faster, more safely and at lower cost than ever before and has led to increased energy production.
Big data has come to stay and is going nowhere anytime soon. So the sooner oil and gas companies start to invest heavily in big data technologies the better for the future of the industry. Bill Gates once said: “The one thing that is different today [in energy] is software, which changes the game.” Investing in big data technology can improve production but companies must consider which big data application will produce the most value before making any investments.