As data volume, variety, and velocity continue to accelerate, power plant operators increasingly encounter situations where 2-D desktop data visualization is insufficient to convey context for advanced analytics, visualization, and collaboration. Using the latest augmented and virtual reality (VR) technologies, combined with advanced analytics, produces solutions with immersive visualizations that can deliver information needed to improve operations, and use data from VR sessions to improve worker productivity.
Virtual reality (VR) and augmented reality (AR) are entering the mainstream in gaming, real estate, design, and more. Headsets, either dedicated or using a smartphone, are readily available, and software frameworks are proliferating. VR and AR have practical applications for industrial operations, from training simulations to maintenance operations. By combining VR and AR with analytics, businesses–including power generators–can optimize operations and maintenance activities to increase productivity and reduce costs.
A good example is the use of VR and analytics for vegetation management. Using asset and geographic data, and LIDAR (Light Detection and Ranging) survey data, virtual models of the vegetation near power lines can be built. Things get more interesting as historical data is added to the model, allowing users to move backward and forward in time to view the growth of vegetation.
Adding machine learning analytical models that can predict the growth of vegetation into the future makes this a powerful operational tool. Virtual tools can be added inside the environment to assist operators in understanding it, such as the virtual measuring stick shown in Figure 1.
 |
| 1. Virtual tools. Utilities often deal with problems created by vegetation near power lines. Machine learning analytical models can predict the growth of vegetation; putting a virtual measuring stick into the model can help show proactively when maintenance will be needed. Courtesy: Space Time Insight Inc. |
Virtual reality helps diagnose problems in dangerous situations. By combining real-time data with an immersive environment and operational controls, the benefits of a live visit are gained without the danger.
Optimizing Maintenance with Augmented Reality
Field technicians may arrive at electrical substations and other work sites without knowledge of the area, and they may also need to find the proper tools and documentation for completing maintenance, tests, and repairs. Sometimes, they need to consult site plans and may need to ask other staff members where assets are located, so they can monitor equipment performance and complete a task checklist. Their hands are often juggling multiple items that must be placed in particular locations to perform a job. Augmented reality is an excellent tool to address some of those challenges, which can improve productivity in real-world maintenance situations.
With augmented reality, field service workers have an overlaid map of the site (Figure 2). Tools and supplies are flagged or highlighted. Digital documents needed for any tasks appear in the technician's field of vision; any chart for an asset appears as the worker looks at the asset.
 |
| 2. Help in the field. Augmented reality (AR) can help workers on a job site. Tools and supplies can be highlighted on an overlaid map, and documents needed to perform tasks can be digitally inserted. AR helps familiarize workers with sites, even if they haven't previously been to them. Courtesy: Space Time Insight Inc. |
Adding machine learning optimization analytics to the mix also increases productivity, by guiding personnel through tasks in an optimal way, as determined by machine learning optimization modeling. Reinforcement machine learning can optimize the order of tasks, note how best to perform each task, and help schedule and route crews to maximize productivity and reduce costs. Humans often exhibit biases that lead to suboptimal performance; using machine learning to find the optimal path, then using VR to train humans to overcome their biases, leads to higher productivity.
In reinforcement machine learning, programmers create a software agent that uses a set of actions to interact with an environment, with the goal of maximizing a target value. The target value can relate to business objectives such as faster processes, lower costs, lower defect rates, or fewer accidents. Over time, the software agent learns what actions to take to alter the environment and maximize the target value.
The software agent isn't programmed to take specific actions in response to specific situations or input; it is programmed with a range of possible actions to take and the ability to observe the results of those actions. Each action the software agent takes generates data about the change in the environment resulting from the action. Each action also generates some level of reward related to the target value.
Over time, the agent discovers which actions generate the most long-term reward by using two approaches: exploration and exploitation. Using the exploration (trial and error) approach, the agent tries an action and then observes the result. When the exploitation approach is applied, the agent uses an action that has worked in the past, then observes how well it works in a new situation. Through these actions, the software agent improves its performance and maximizes its target value.
To apply exploration and exploitation, reinforcement machine learning needs ways to observe and influence its working environment. This isn't a trivial exercise when it comes to maintenance operations, especially for assets and locations that don't require frequent maintenance. While it is possible to develop these models with GPS tracking, careful logging, and a variety of sensors, it would still be a challenge to generate enough data–enough exploration–to refine the model. Virtual reality presents a unique solution.
Immersive Visualization: Generating Data to Drive Analytics
Virtual reality simulation has long been used for training, allowing personnel to tackle complex and at times dangerous tasks in a controlled, safe environment. It allows jobs to be mocked-up and observed, so potential mistakes can be identified and corrected.
VR also presents a unique opportunity to "analyze action." It can help users understand human perceptual constraints. For example, an environment can be created and what workers notice and what they miss can be analyzed. The environment can then be modified and more trials run. This reduces the time required to design safety alerts and protocols, and then train workers to recognize and follow them. In a sense, VR can be used to develop "muscle memory" for safe practices.
In a properly instrumented VR environment, a stream of data can be produced on which to perform analytics. In a training environment, the user's activity stream can be analyzed and checks can be performed to ensure all steps in a protocol were completed correctly. Efficiency can be analyzed. How many steps did it take the user to perform the task? Did they select the right tools? Did they perform unnecessary tasks? A performance score or a safety score can be calculated, and a user can be compared to various norms, accounting for experience, role, even geography.
A potential pitfall is that not everyone performs optimally in a VR environment. Differences based on past VR experience or even a tendency for some people to get nauseous in a VR simulation need to be considered.
These performance analytics can help in the development of a machine learning model to predict what an individual's score will be in the simulation. By combining simulation performance data with demographic and other employee data, a sophisticated model that can predict who will perform well on any given set of tasks can be developed. It might also be used as a screening tool to understand which new hires are likely to succeed in a given role.
VR and AR can be used to train personnel to perform a given set of tasks in an optimal way, but to develop the machine learning model, developers need to understand all the different ways a task could be performed. One way to get that data is through VR training simulations.
The machine learning models need data such as how long it takes to perform a task, the starting locations of equipment and tools, and any constraints on performance such as safety protocols, union rules, and availability of personnel. It may be possible to collect this data over time from real-world maintenance activities, but VR simulations allow collecting far more data in a shorter period of time, and environmental parameters can be managed very quickly. In the real world, it's impractical to repeatedly relocate assets, change signs, or reorganize a truck's tools; in VR, almost anything is possible.
The data generated by the simulation is fed to a path analytics model that will find the optimal path through a set of tasks. An example of a path analytical model that may be familiar to readers is the "ant colony algorithm." The process was developed based on the method ants use to find an optimal path to food–relying on exploration and exploitation.
As previously mentioned, a VR simulation can be used to understand the results of taking various paths through a series of steps. However, if a path analytical model is to find the optimal way to perform tasks, exploration must also be encouraged.
Gamification might be one way to provide the needed incentive to keep users looking for the ideal way to complete a simulation. Gamification is, as it sounds, a way to turn a set of tasks into a game. For example, points might be assigned to a given way of performing a task or for taking certain steps in a certain order. The points awarded could be reduced as more users perform steps the same way (exploiting the discoveries of others). More points could be awarded for trying something new, which would encourage exploration. As the model matures and exploration is no longer producing gains such as faster times or lower costs, the point system could shift to reward compliance with the optimal model.
This two-way, iterative approach to combining VR and analytics should be a potent operational optimization tool. VR can help managers understand present operations and train employees in current protocols. It can then produce data that ultimately enables improvement of operations and protocols. Operators can train employees in the new methods, creating a circle of operational improvements.
Virtual reality provides the context to use big data to make fast, confident decisions. Combined with advanced machine learning analytics, immersive environments become a powerful tool for optimizing operations. Asset-intensive organizations such as utilities, transportation networks, and manufacturing will use virtual and augmented reality to understand and take action on their data to improve the performance of their operations and assets. ■
–Jeff Weitzman leads marketing and product strategy at Space Time Insight Inc.