The challenges of storing big data in smart cities

While cities have long depended on video surveillance to enhance public safety and deter crime, today’s smart cities are harnessing those solutions for a multitude of additional purposes, including traffic management, lighting, parking enforcement and much more. 

This is due in large part to the fact that cameras themselves have become intelligent computing devices that offer video analytics and the ability to integrate with disparate systems and devices. Among the many potential applications for which surveillance and other data can be combined for smart city applications is transportation. By deploying surveillance cameras with object detection and recognition analytics, officials can automatically detect stalled cars, wrong-way drives and other obstacles that can impact both safety and travel times. Using this insight, cities can use traffic management and/or digital signage systems to alert drivers to potential issues and reroute them to avoid these obstacles. 

The larger goal of smart cities is to create efficiencies, improve sustainability, create economic development and enhance overall quality of life factors for those who live and work in the city. To do this, municipalities can use transportation data collected over longer time frames to identify and alleviate routine bottlenecks and other issues that negatively impact travel times and overall quality of life. 

This is just one example of how municipalities can implement smart city platforms and technologies to achieve these goals. At the heart of these systems are the solutions used for storing all this data. Below are some key factors to consider when evaluating these solutions, which are the foundation for smart city applications.

Tying it all together

According to IHS Markit, the amount of data generated worldwide by surveillance cameras alone reached 2,500 petabytes per day in 2019. Add to that data generated by traffic management, access control, public transportation and much more and smart cities are looking at big, big data, all of which has to be collected, aggregated and stored for the deep analysis that leads to the kind of insights that cities can act upon to improve safety, efficiency and the everyday lives of residents. While all this data is valuable in its own right, the true value lies in the ability to aggregate and analyse data from disparate sources to deliver deeper insights and provide a fuller picture of overall city operations.

Research firm IDC has predicted the cumulative amount of data generated worldwide will grow from 33 zettabytes (33 trillion gigabytes) in 2018 to 175 zettabytes by 2025. As evidenced by the IHS Markit data above, much of this data will come from IP surveillance cameras for both security and business intelligence purposes. As a result, in many cases, the video surveillance system offers an ideal solution for tying all this information together. While enhanced by artificial intelligence (AI), today’s surveillance solutions are capable of providing the kind of actionable insights and intelligence smart cities require. 

At the heart of these applications are surveillance storage solutions, which are capable of aggregating all this data both at the edge for immediate analysis and real-time intelligence as well as in back-end environments for long-term analysis, overall trends and deep learning analytics that make the system smarter over time. These robust solutions are built to withstand the rigors of smart city applications and deliver a number of other features that deliver benefits for municipalities.

Finding the best storage fit

In the age of the IoT and AI, where devices are generating more data than ever before, deciding how to properly store all this data has become a critical consideration. Depending on a city’s needs, there are several approaches available, including storage at the edge (close to where the video or data is captured), in the cloud (a large, centralised backend storage server or servers) or even a combination of both, which some experts are calling IT 4.0.

For smart cities with hundreds or thousands of cameras and IoT sensors, cities should take a multiple-layer approach to storage architecture that delivers the benefits of both edge- and cloud-based storage. 

The main reason for this is that streaming large amounts of video and data to the cloud is expensive and can suffer from significant latency issues. The solution for smart cities is to deploy technologies that aggregate, filter and analyse data at endpoints and at the edge and then have relevant alerts sent to the video management system at the head-end for review and response. This allows data to be quickly processed and notifications to be delivered, in a more timely manner, which can improve public safety. 

For example, when a municipal camera detects an incident using video analytics or AI, it can send an alert to a monitoring or command centre, allowing a police officer to respond quickly. An NVR appliance built with a surveillance-optimised hard drive allows dispatchers or operators in the command centre to access video of the incident to further assess the situation, enabling them to provide the officer with real-time situational awareness and actionable intelligence about the incident as he or she is en-route to the location. 

For archival and analysis to provide deeper trends, video and data can then be uploaded to the cloud, where providers utilise high-quality enterprise and solid-state drives to ensure that video and data from a city’s IoT devices will be protected with the highest level of integrity. In this centralised environment, weeks or months of video and data gathered from traffic cameras, for instance, can be analysed to identify peak traffic hours and traffic patterns throughout a city, allowing officials to make decisions that can improve people’s commutes, such as synchronising traffic lights at certain intersections.

While this multiple-layer approach is ideal, there are situations where it’s just not feasible for a city. Internet speed and the distance between camera or sensor and server are among the factors that may be a hindrance to cloud-based storage. Slow speeds and longer distances may make uploading or streaming files take a long time. Additionally, monthly subscription fees for cloud storage can also be an impediment to cloud-based storage for some cities. Therefore, the storage model a city ultimately chooses depends not only on the goals of the application but also infrastructure and budget.

Technology drives robust smart cities

As the sheer number of IoT devices and smart surveillance cameras deployed in the field increases, there is much greater stress on storage solutions. They must not only record, process and analyse video and other data, but they also must perform these tasks on a 24/7 basis. These solutions must also be able to accommodate the growing number of smart city applications that rely on deep learning. Therefore, it’s important to deploy surveillance- and AI-optimised hard drives tuned for around-the-clock workloads that deliver superior read and write capabilities.

Surveillance-optimised hard drives are capable of withstanding excessive heat, stress and vibrations that often occur in a storage system. These specialised drives can proactively prevent malfunctions to avoid potential data compromise. 

Given the sensitivity of smart city data and the potential ramifications of a network breach, cybersecurity is a paramount concern. Naturally, this means that any camera, device or sensor connected to the network must offer top-of-the-line cybersecurity measures to prevent hackers from using them as a back door to breach the network and gain access to data using default passwords or other vulnerabilities. On top of this, data can also be stolen from hard drives themselves due to improper disposal. To combat that possibility, it’s important to select drives that offer hardware-based encryption as well as secure methods for erasing data on drives that have reached end of life or have been taken out of service for any number of reasons. These features provide both integrators and municipalities with peace of mind that they can avoid this type of breach and the consequences that would ultimately follow. They also ensure compliance with standards such as TAA and FIPS.

Because smart city applications are 24/7 operations, storage solutions must be functioning optimally at all times. Unfortunately, drives do fail from time to time, so it is vital for cities to prepare for that possibility. Embedded health management software monitors and analyses drive operating conditions at all times. Using advanced algorithms, these solutions detect anomalies in temperature, vibration and more to limit deterioration and prevent drive failure. By implementing these solutions, municipalities can maximise the health and workload performance of their drives to create more efficient storage solutions that deliver better data insights for improving city operations.

However, even with the most robust health monitoring software solutions in place, there may still be times when unexpected events can cause drive failure. To prepare for just such occurrences, municipalities should ensure there is redundancy in their systems and augment that by subscribing to data rescue or recovery services, which allow data to be recovered for up to two years in some cases. These types of services are especially crucial for law enforcement or other public safety personnel who require data to be stored for lengthy periods of time.

Conclusion

There is no doubt that smart city applications offer the ability to improve operations in a number of areas, not the least of which is quality of life and storage solutions can provide a strong foundation for these initiatives. By following the above advice, cities can choose storage solutions that will offer high performance and reliability and ensure they are deploying it in the most appropriate way, both of which are key to determining the success of any smart city application. 

By Jessica Burton, Global Surveillance Product Marketing Manager, Seagate Technology

www.seagate.com