2017 Audit of Topaz Solar Farm

2017 Audit of Topaz Solar Farm
March 2018
STAFF REPORT
PREPARED BY: ELECTRIC SAFETY AND RELIABILITY BRANCH
SAFETY AND ENFORCEMENT DIVISION
Edmund G Brown, Jr., Governor

Audit Report of Topaz Solar Farm
Table of Contents
I. Introduction ................................................................................................................................. 3 II. Findings Requiring Corrective Action ....................................................................................... 3
Finding 1 – Topaz fails to address SCADA issues that may hinder reliable plant operations. . 3 Finding 2 – The Plant‟s contractor management program is inadequate................................... 4 Finding 3 – The Plant fails to manage vegetation to minimize fire risks. ................................. 5 Finding 4 – The Plant fails to complete work on inverter fault detection and isolation to
prevent grass fires. ......................................................................................................... 6 Finding 5 – The Plant fails to maintain warning labels on combiner boxes and confined
spaces. ............................................................................................................................ 7 Finding 6 – The Plant lacks evacuation maps at power blocks having alternate muster point.. 9 Finding 7 – The Plant fails to post evacuation maps at each exit doors at the O&M building.
...................................................................................................................................... 10 Finding 8 – The Plant lacks hazmat placards in designated areas. .......................................... 11 Finding 9 – The Plant‟s work planning document conflicts with actual practice. ................... 13 Finding 10 – The Plant fails to check engine coolant and oil levels on a weekly basis. ......... 15 Finding 11 – The Plant fails to follow-up on a contractor‟s recommendation. ....................... 15 Finding 12 – The Plant uses 3-way Communication and Peer Checking in Lockout Tagout
(LOTO) Procedures instead of Independent Verification. ........................................... 15 Finding 13 – The Plant lacks a Continuing Training Program ................................................ 16 Finding 14 – The Plant‟s CPUC Reporting Procedures required updates. .............................. 16 III. Observations and Recommendations ...................................................................................... 17 Observation 1 – The Plant‟s spill kits lack identification and labeling. .................................. 17 Observation 2 – The Plant lacks a written procedure for use of company vehicles. ............... 18 Appendix – CPUC-Requested Documents ................................................................................... 19
Page 2 of 20

Audit Report of Topaz Solar Farm
I. Introduction
This is the 2017 Audit Report of the Topaz Solar Farm (“Topaz” or “the Plant”) prepared by the California Public Utilities Commission‟s (“CPUC‟s” or “Commission‟s”) Electric Safety and Reliability Branch (ESRB). ESRB audited the Plant for compliance with Commission General Order (GO) 167, which includes Operation, Maintenance, and Logbook Standards for power plants. GO 167 requires generating asset owners to operate and maintain their power plants in a safe and reliable manner. Electricity is vital to the State‟s economic well-being and the safety of its residents. Therefore, ESRB enforces GO 167 and conducts compliance audits to ensure electric generation safety and reliability for the State.
On September 15, 2017, ESRB notified Topaz of the audit and requested pertinent documents that include operation procedures such as lock-out tag-out, confined space entry, and emergency response procedures, as well as maintenance records. ESRB‟s initial data request is included in the Appendix. Prior to the start of the onsite audit, ESRB reviewed the submitted documents and requested additional follow-up data. ESRB conducted the onsite audit from October 23 through 26, 2017 during which it observed plant operations, inspected facility and equipment, reviewed additional data, and interviewed plant staff. During and after the onsite audit, ESRB requested and reviewed more data. From these activities, ESRB evaluated whether the Plant 1) complies with GO 167 requirements, and 2) could improve its programs, procedures, and policies to enhance safety and reliability.
ESRB identifies 14 findings in Section II, which represent potential violations of Operation and Maintenance Standards. These deficiencies can adversely affect reliable operation and present safety hazards to plant staff. ESRB also notes two observations and recommendations in Section III.
II. Findings Requiring Corrective Action
Finding 1 – Topaz fails to address SCADA1 issues that may hinder reliable plant operations.2
Topaz fails to address and resolve Supervisory Control and Data Acquisition (SCADA) issues that may affect reliable plant operations. Topaz uses the control software called Wonderware that serves as the Human-machine Interface (HMI3) and supervisory control of its power blocks. While Topaz is remotely-operated from Tempe, Arizona, the Plant does have a complete redundant control room onsite. ESRB toured the control room and received a demonstration
1 SCADA is a system of software and hardware that allows a plant to remotely operate and control equipment. It also enables a plant to acquire and store key operating data of the equipment, particularly to analyze their performance. 2 GO 167 Operation Standard (OS) 4 - Problem Resolution and Continuing Improvement, OS 8 - Plant Status and Configuration 3 HMI is the graphical display that interfaces between the human operator and the equipment/machine being operated and controlled. Through the HMI, an operator can see schematics of the system on computer screens and can operate remotely-controlled equipment, e.g. open or close switches, turn on or off motors/pumps, and modulate valves, etc.
Page 3 of 20

Audit Report of Topaz Solar Farm
from an operator, see Figure 1. ESRB noticed several software issues in the HMI that may impact plant operations. In particular, ESRB saw the HMI showed a graphical display of an opened circuit breaker while in fact the breaker was closed at the time according to the operator. ESRB also observed an indicator on the HMI flashing on and off intermittently. The operator explained that the flashing alert indicates a loss of communication from the inverter, though it‟s attributed to a glitch in the software rather than an actual loss of communication. ESRB inquired about these glitches and discovered that the Plant has known, but failed to repair, at least a dozen of these so-called SCADA issues. Some of these include nuisance alarms that may be caused by improper setpoints. Though Topaz is two years into commercial operations, it has yet to resolve these issues. The issues that ESRB noticed in particular can cause an operator error and may hinder reliable operations. Topaz must evaluate and take proper corrective actions to correct these SCADA issues.
Figure 1: Topaz‟s SCADA operator control room.
Finding 2 – The Plant’s contractor management program is inadequate.4 Topaz lacks an adequate contractor management program. The Plant‟s current program to manage contractors consists of only a simple flow chart.5 While the chart explains that Topaz assesses risks thoroughly when evaluating a contractor for projects over $2 million, the chart lacks sufficient detailed information on how Topaz actually prequalifies, selects, and oversees contractors to ensure worker safety and work performance. Topaz states that it is currently
4 GO 167 Maintenance Standard (MS) 1 - Safety, OS 1 - Safety, and OS 7 - Operations Procedures and Documents 5 Topaz‟s Risk Assessment (Prequalification) Process Flow Diagram, FS.462018, Version 0, Last Updated: April 11, 2016
Page 4 of 20

Audit Report of Topaz Solar Farm
formalizing a program. However, the program is still in draft form.6 Further, ESRB reviewed a list of prequalified contractors and found at least one contractor (Electricraft) with a high Experience Modification Rate (EMR) subjecting it to additional review per the Plant‟s impending program.7 A high EMR means a company has a higher-than-industry-average number of worker injuries and compensation claims. It indicates a company may be unsafe. Even so, ESRB found no evidence that shows the Plant performed a secondary review, which further illustrates the need for a robust contractor management program.
Topaz must adopt a robust contractor management program to ensure the proper oversight, qualification, and safety of contractors.
Finding 3 – The Plant fails to manage vegetation to minimize fire risks.8
Topaz fails to manage vegetation to minimize fire risks. During the onsite visit, ESRB observed overgrown vegetation of at least several feet tall, some of which encroaches onto the solar panels, see Figure 2. This is contrary to the Plant‟s vegetation management goals to “manage fuel load to minimize risk of fire” and to “manage vegetation height to maintain optimum function of solar PV arrays”.9 Topaz is subject to environmental restrictions and maintains at least eight inches average height and at least 500 pounds per acre residual dry matter to promote a natural habitat for the San Joaquin Kit Fox, an endangered species.10, 11 Further, Topaz is restricted from using motorized trimming and/or mowing onsite to protect wildlife except along roadsides, fence lines, and other obstructed areas.12 The Plant primarily uses sheep and cattle grazing as a mean to vegetation management. However, as evidenced by the overgrown and dry vegetation in some areas, this method proves ineffective in mitigating potential fire risks. The Plant is situated in the Carrizo Plain, an area of arid climates where low humidity and strong winds, combined with dry grass, make it highly susceptible to wildfires. Additionally, the Plant has had a history of fires in and around the solar panels. Since commercial operations in February 2015, Topaz has had 12 fires.13 In four occasions, an avian shorted an overhead conductor on a riser pole. The bird was electrocuted, caught fire, and fell to the ground, where it ignited dry vegetation and started a fire. In eight separate occasions, a rodent chewed on an underground cable, which shorted a circuit and caused a ground fault resulting in a fire. So fires have occurred and do occur at Topaz necessitating prudent and effective means of managing
6 Topaz‟s Subcontractor Management Program document, Revision 0 7 Electricraft had an EMR of greater than 1.0 in 2010, 2011, 2014 and 2015. 8 OS 1 - Safety, OS 4 - Problem Resolution and Continuing Improvement 9 Topaz Solar Farm Vegetation Management Plan [Revision February 8, 2012], Section 1.5 Topaz Vegetation Management Goals, at p. 8. 10 Topaz Solar Farm Vegetation Management Plan [Revision February 8, 2012], Section 3.5 Grazing Intensity and Management, at p. 39. 11 U.S. Fish and Wildlife Service‟s Biological Opinion dated July 22, 2011, at p. 15, “The ground under and between the solar arrays will be vegetated with a dominant cover of grasses and herbs to promote a more natural habitat that supports kit fox prey. Vegetation height will be controlled by pulse grazing or other means not hazardous to kit fox. A managed grazing regime will be implemented to maintain vegetation height suitable for kit fox use and their prey”. 12 Topaz Solar Farm Vegetation Management Plan [Revision February 8, 2012], Section 1.7 Protection of San Joaquin Kit Fox and Nesting Birds and Their Habitat, at p. 9. 13 Topaz fire incidents with causes and corrective actions.
Page 5 of 20

Audit Report of Topaz Solar Farm
vegetation to reduce fire risks. Topaz must evaluate and implement effective alternatives to better manage vegetation to minimize fire risks.
Figure 2: ESRB observed overgrown vegetation of at least several feet tall, some of which encroaches onto the solar panels.
Finding 4 – The Plant fails to complete work on inverter fault detection and isolation to prevent grass fires.14 Topaz experienced 12 grass fires since July 2015: two in 2015, seven in 2016, and three in 2017. The majority were incipient fires that were contained by plant staff with fire extinguishers before CalFIRE arrived. Topaz identified that the fires were caused by avian contact with overhead lines and ground faults resulting from rodents chewing on low voltage circuits. The Plant also identified solutions to address each of the causes. For example, the Plant installed customized “green jackets” on overhead conductors to prevent avian contact in February 2016, see Figure 3. The Plant also implemented automatic ground fault detection and isolation in one type of inverter in late 2016 (Topaz uses two types of inverters onsite). Both solutions proved effective and the number of fires was significantly reduced from 2016. And Topaz has not had any further fires from the enhanced inverters in 2017. However, Topaz has failed to implement automatic fault isolation on all inverters. The Plant must complete installation of this protection scheme on all inverters to reduce fire risks.
14 MS 1 - Safety, OS 1 - Safety, and OS 20 - Preparedness for On-Site and Off-Site Emergencies
Page 6 of 20

Audit Report of Topaz Solar Farm
Further, as part of its overall fire mitigation, Topaz must coordinate with the local fire department at the next emergency drill to address how to best to alert its neighbors who are in close proximity of the Plant in the event of a fire emergency. Topaz must critique its drill and incorporate any lessons-learned in its Emergency Action Plan.
Figure 3: The Plant contracted with Cantega Technologies to install custom-designed “green jackets” shown as 201 and 202 to prevent avian contact with energized overhead conductors. Finding 5 – The Plant fails to maintain warning labels on combiner boxes and confined spaces.15 In Block 5, ESRB observed faded, partially-peeled, or otherwise illegible warning labels. For example, on some combiner boxes, the arc flash warning labels as required by National Fire Protection Agency (NFPA) 70E16 have either partially or completely peeled-off, rendering them unreadable and ineffective, see Figure 4.
15 MS 1 - Safety 16 NFPA 70E is the standard for electrical safety in the workplace. Article 130.5(D) and 130.7(E)(1) [2015 version] sets the standard for arc flash labeling and best practices for maintaining such labels.
Page 7 of 20

Audit Report of Topaz Solar Farm
Figure 4: Partially-peeled and faded arc flash warning labels on a combiner box. Topaz must inspect warning labels on combiner boxes throughout the plant and replace them as necessary. Arc flash labels should withstand their usage environment. If necessary, the Plant should consider applying an over-laminate to protect the printed surface from exposure to sunlight and other elements. ESRB also observed faded confined space labels throughout the plant, see Figure 5. While Topaz‟s confined spaces are classified as “non-permit” spaces, they, nevertheless, have limited or restricted access and may otherwise present a safety hazard to workers. Topaz must inspect warning labels on confined spaces throughout the plant and re-stencil them as necessary. The Plant should consider installing Occupational Safety and Health Administration (OSHA)-compliant signs that are more visible and resistant to wear, see Figure 6.
Page 8 of 20