Pump Test Station Used Across Multiple Locations Worldwide
Simplifies gathering test data from multiple plants to perform site-to-site comparisons
Client – Large Pump Manufacturer
Our customer needed an updated test system to replace their obsolete/unmaintainable test systems. It was too hard to gather data from multiple sites located around the world, certain algorithms weren’t standardized, and they didn’t have the ability to utilize the test systems to calculate first-pass yield at each site.
Viewpoint developed a new pump test application that harmonizes the user interface, calculations, and test procedures, resulting in enhanced operational efficiency and tracking of the manufacturing process. Furthermore, it is used in multiple global locations with language localization and support for the varying hardware already in place at each site. The new application deposits all the test data from the various sites into a single database for engineering and manufacturing data analysis at corporate headquarters.
Simplifies gathering test data (for analysis) from multiple plants worldwide to perform site-to-site comparisons.
Enables calculation of first-pass yield for each manufacturing plant.
Standardizes algorithms for pump performance summary data calculation for first-pass yield and site comparison analysis.
Abstracts the hardware to support differences in control and measurement hardware at each site and provides a future path to homogenize hardware installations.
The entire system consists of three main applications:
A database management application to allow the user to update test configurations and to enter and associate new pump serial numbers with specific tests.
A pump test data acquisition application that can run up to 5 different tests simultaneously on the UUT, generating a datafile for each test that is also stored in the database.
An application to generate reports from the information stored in the database to provide to customers when delivering a new pump or after a factory witness test.
Custom Endurance Test System – for a medical device
Increased level of automation allows for multi-day and multi-week test runs
A medical device manufacturer
Our client wanted to improve the endurance testing of an implantable medical device product to help determine the recommended lifetime of the product. An obsolete test system existed, but the client wanted improved performance, UX and configurability. They wanted to just hit the “go” button and let it run for days or weeks. They also needed to be able to have new features added after the first release.
The custom product validation endurance test system utilizes NI cDAQ off-the-shelf hardware combined with custom LabVIEW-based software to provide automated N-up endurance testing of the UUT.
Higher fidelity DAQ
Increased configurability of the system to run tests the way the client wants to
Increased level of automation allows for multi-day and multi-week test runs.
The endurance tester physically stresses the UUT to measure force and eventually breakage events. These events are used to help determine the recommended lifetime of the product. The tester tests multiple UUTs in parallel in order to gather more data faster for statistical validity. The system collects data until all UUTs break or the operator stops the test.
Viewpoint provided the software and advised DAQ hardware selection. The rest of the test system hardware was selected and assembled by the client.
The automated test system applies a varying cyclical force to multiple UUTs while measuring the force applied to the device. The software automates the data acquisition, analysis, load application, and motor during a test. The system measures all forces applied simultaneously while synchronizing that data to a cycle counter. That data is analyzed to determine average, maximum, and minimum force applied to the device over a user configurable number of cycles.
While running there are multiple alarm states that are monitored. When these alarm states occur, a file can be generated to dump a user configurable duration of force measurements to a file. Other alarms generated trigger the system to change a digital output state triggering a text message to be sent to the operators of the system. The system was designed to test for weeks at a time.
Our client was already doing validation, but it was manual, and the client’s customer started requesting faster turnaround of results. Their customer was also requesting data to be sent with the results. Our client chose to automate the validation process to enhance their productivity.
Logs errors during the test (e.g., for continuous monitoring tests, logging the number of instances of when a UUT’s LIN (Local Interconnect Network) response deviates from a static, current draw outside of limits)
Capable of testing a large variety of product lines
Logs pertinent data to a database for post-test analysis/inclusion into reports
The UUT is an electro-mechanical part that falls under a variety of different product lines. As such, the client had a couple variants of the tester, based on the communication needs of the UUT. A total of more than a dozen testers were deployed. The functionality of the tester evolved over time, specifically modifying software to make the tests faster / decrease cycle time.
Extensive diagnostic/manual operation of system for debug of software and electrical connections between the UUT and the test stand/tooling.
Product-specific software components to operate unique products.
Execute mechanical endurance tests.
Execute environmental endurance tests.
Database output containing results from every test cycle (either mechanical cycles or time depending on test being run).
The client already had a test system in place, but it was old and was becoming unmaintainable. Increasing demands from the test engineers and the old software architecture not lending itself to clean implementation of these new features (new sequencer capabilities and ECU CAN communication) drove the need for a rewrite of the software application.
The updated product validation tester supports product validation of the UUT by automating long tests (sometimes a week or more) providing the desired set point control, allowing the client to prove more obviously that their part met the stated specification. Viewpoint developed the software and the client selected the hardware.
Automate long duration tests
Improved operator UX by making controls and indicators more intuitive to the user as well as providing additional capability within one application.
Acquire ECU data along with measured UUT data to allow for engineering performance characterization analysis
Playback utility enables the Test Engineer to quickly view collected data to chart out a path forward for further testing.
Automate a Design of Experiments matrix of conditions, through new sequencer capabilities, to more quickly arrive at product characterization parameters.
All collected signals are now housed in one TDMS file instead of multiple files from different applications.
The UUT is a complete engine with a focus on one of the mechanical subsystems. Data is collected on over 100 channels, measuring temperature, vibration, strain, RPM, position and pressure. Engine management data (e.g., component location, pressures, engine speed, and status flags) is collected via CAN. The engine speed is set via an analog output, and subsystem setpoints are sent to the ECU via CAN. SCXI still used on some of the old test stands, but is being phased out in favor of cDAQ. The test system software was developed in LabVIEW.
An automated system permits faster validation, unattended test, an increase in throughput, and can free up resources for other tasks during the weeks long endurance test.
Client – A manufacturer of aircraft components in the mil-aero industry
New product development drove the need for a new endurance test system for product validation. The old systems were not designed to test the newly designed part (aircraft actuators), and the company didn’t have the time or resources to reconfigure existing systems to perform the testing required.
The new PXI-based endurance test system provides automated electromechanical testing, full data recording, report generation and a diagnostic panel for intelligent debug. Viewpoint selected the NI equipment, while the test consoles, and other components were selected and fabricated by the customer.
An automated system permits faster validation, unattended test, an increase in throughput, and can free up resources for other tasks during the weeks long endurance test.
Full data recording with a data viewer enables post analysis, which provides the ability to review and analyze raw signals captured during execution. Channel examples are actuator LVDT position, load, current, and encoder actuator position.
Summary report capability allows the customer to document the amount of testing completed against the full endurance test schedules.
A manual diagnostic operational panel provides the ability to verify particular DUT functionality or components without running an entire schedule.
Systems can be paused and restarted to allow for “scheduled maintenance” of the DUT such as inspections, lubrication, etc.
The PXI-based endurance test system enables data collection, deterministic PID Loop Control, emergency shutdown and a diagnostic panel for manual test and debug operation. The system runs endurance test schedules, that are defined as a recipe for test execution. These schedules, which are customer-defined and DUT-specific, are designed to simulate the actual conditions the DUT would see in real world application as closely as possible. LabVIEW-RT was used for the deterministic looping for Closed Loop Control of Actuator Position and Load Control. LVDT demodulation was performed on a PXI FPGA card programmed with LabVIEW FPGA.
Full Data Collection for Real-Time and Post Analysis
Deterministic PID Loop Control
Diagnostics Panel for Manual Test and Debug
Endurance Test Schedule Execution
Hydraulic Control Panel for Source & Load PSI Control
Ability to run tests unattended and overnight reduces operator labor and compresses test schedules
Client – Major Aerospace Component Supplier / Manufacturer
The client had an older VB & PLC-based test system in place already, but it was obsolete. A new endurance test system needed to be developed to validate prototyped components (in this case, aircraft & aerospace bearings). Many of the prototypes are one-off, so it was important that the test system not destroy the component.
A new endurance test system was developed to validate prototyped components. The test system can be configured for automatic shutdowns so as not to destroy the component under test in the event of unexpected performance of electro-mechanical subsystem components. The updated endurance tester supports product validation by allowing the product to run under various test conditions (e.g. speed, load, oil flow, temperature) and collecting data for analysis.
Viewpoint developed the software and selected the NI hardware (other hardware was selected by the client).
Ability to run tests unattended and overnight eases operator labor and compresses test schedules
Data collection allows for offline engineering analysis
Automatic shutdowns reduce destruction of the prototype component under test
The updated cRIO-based endurance tester incorporates configurable profiles, data logging, and automatic shutdown to allow for safer extended validation testing. LabVIEW FPGA and LabVIEW RT were used together to interface with the test hardware sensors and controls. LabVIEW as used create the HMI for the test system.
Closed loop control of bearing test oil flow
Axial load control
Driver for Emerson VFD
E-Stop and safety management (shutdowns based on alarm limits)
Data collection – temperature, pressure, flow, vibration, frequency
Multiple International Deployments Helps Prove Product Meets Spec.
Each endurance test can run upwards of 6 months.
Client: Major Automotive Component Supplier
A new endurance test system was developed to give more precision in the control setpoint. This additional precision enabled potential clients to review the product performance in real-life situations. Each endurance test can run upwards of 6 months.
The updated endurance tester supports product validation by providing the desired parameter control method, allowing the client to prove more obviously that their part met the stated specification.
Viewpoint developed the software and selected the NI hardware for the first unit. The client is now deploying copies of this system to multiple international manufacturing plants.
Able to prove meeting a particular product specification of interest
Closed loop parameter control
Emergency shutdown functionality
The cRIO-based endurance tester provides closed loop control, data collection, and alarming with controlled and emergency shutdown functions. The operator can manually configure a test or load a saved configuration. After a manual operator check to make sure the setup is operating correctly, a successful test will run its full duration and stop on its own.
Product Validation using LabVIEW RT & LabVIEW FPGA – An electromechanical actuator test stand
Automated testing reduces operator man hours and increases production throughput.
Client – A manufacturer of actuators in the mil-aero industry.
New Product Introduction (in this case a new controller and new actuators) drove the need for a new automated electromechanical test stand.
New NI PXI-based electromechanical test equipment provided automated testing, report generation, and SPC data generation. The sequencing of the test procedure, reporting, and verifiable results were managed with the StepWise platform.
Automated testing reduces operator man hours and increases production throughput.
Meets strict customer requirements regarding testing and data recording in a verifiable manner.
Automated Test Report Generation.
Collects data to support SPC (Statistical Process Control).
Ability to obtain the internal state of the controller FPGA via the LVDS communication link.
Viewpoint developed the software and selected NI data acquisition and control hardware for the test stand. There are several layers of software functionality.
HOST LABVIEW SOFTWARE LAYER
Test steps (e.g. Frequency Response, Step Response, Dynamic Stiffness, Fault Response, Power Consumption)
Test Report Generator
REAL-TIME (RT) LABVIEW SOFTWARE LAYER
LABVIEW FPGA SOFTWARE LAYER
Synch data from 3 sources (tester, UUT, external DAQ device)
Stream high-speed data to disk
Stream high-speed data to analog outputs for engineering use
Custom communication protocol used by UUT over LVDS lines
NI FlexRIO card with LVDS adapter module
Multiple NI R Series cards
High speed, high voltage, isolated analog input cards
It did not provide ability for unattended operation
The thermal control had to be set manually
They wanted to do less manual review of the data
The client develops mission-critical products, so there’s a desire to reduce manual operations because they have to explain any anomalies, and manual operations are typically more error-prone. They needed repeatable results that they could trust.
Viewpoint developed a new test system that utilized new hardware and software, augmented by existing low level hardware and firmware. The test system was developed to perform both functional test for production and environmental testing, and was designed to handle up to 4 DUTs at once. The test system utilizes the StepWise test executive software with custom test steps, which allowed the client to create their own highly configurable test sequences. The system was developed in two phases, with the second phase adding support for a FPGA expansion backplane (NI CompactRIO chassis) in order to provide future capability for bringing some of the microcontroller sequence activity into the NI space. In addition, the previous version had a mix of serial, TTL, and USB instrumentation, which was not as robust as Ethernet based instrumentation. Phase II involved upgrading to all Ethernet based instrumentation, and did away with the original test system’s many manual toggle switches that could be used instead of the programmable mode through the SW.
~40% test time reduction per unit
~25% reduction in anomalies that needed to be justified
Designing an Automated Fuel Cell Validation Test Stand
Verifying a New Fuel Cell Design Through Automated Operation
Client: A major automotive manufacturer
Micro Instrument, an automation vendor that builds test and validation stands, has extensive experience with programmable logic controllers (PLCs) and stand-alone controllers for controlling repetitive motion, safeties, and other “environmental” parameters such as pressure and temperature. The company typically uses PLCs to reliably deliver discrete I/O control and standard PID loop control.
However, Micro Instrument’s customer, a major automotive company, was interested in investigating fuel cells as a power source and they needed to run these fuel cells under a wide range of conditions for extended durations, for both design validation testing and durability testing purposes. Furthermore, the client wanted to implement more advanced control algorithms than simple PID.
The customer knew they needed control loops that predicted system response so we could eliminate overshoot and/or achieve a faster approach to a setpoint. But, because the customer did not know in advance exactly what such “smart” controls would entail, it was beneficial to have the full power of LabVIEW to develop such controls. Providing this functionality with a PLC would be cumbersome, if not impossible.
The customer had some Compact FieldPoint which they wanted to use for this project, so we needed to ensure that this equipment would be sufficient to deliver the required control performance and tolerances. Also, the system needed to conduct PID control in two forms – PWM and continuous control. Importantly, this Fieldpoint hardware had a real-time controller running LabVIEW Real-Time.
We developed a flexible control environment using NI Compact FieldPoint and LabVIEW Real-Time to meet the customer’s system control demands. For example, to predict system response, we programmed the Compact FieldPoint to run control loops that were aware of imminent system-state changes and changed their control schemes accordingly.
As with most validation test systems, we needed to monitor conditions for safety. New product designs are often operated near the edges of safe operation in order for the designer to understand how the product performs in extreme conditions. For this fuel cell application, destructive over-heating and over-pressure could occur. Both digital and analog signals were watched in real-time to assure operation within reasonable bounds and allow a safe shutdown if the fuel cell ran into out-of-bound conditions.
The application used the following independent parallel loops:
Seven for PWM-based temperatures control
Two for continuous pressure monitoring
Four for solenoid and sensor monitoring and control
15 safety loops
Data collected during the validation tests were saved to a local PC for later performance analysis and anomaly detection.
The combination of Compact FieldPoint with LabVIEW Real-Time enabled the customer to run the required custom control algorithms and it surpassed the capabilities offered by standard PLCs.
Monitoring of Testing Inside Environmental Chambers
Our customer required a system that would replace manual charting of tests performed inside various environmental chambers.
Viewpoint designed an automated solution which notifies the technician when the test chamber requires attention and reports chamber utilization for planning and scheduling purposes.
This application was designed for a group that provides long-term thermal and environmental testing to a large number of internal customers at its facility. The group is responsible for approximately 80 environmental chambers which are used for a variety of tests for electronic circuit boards and modules.
These tests typically last between 100 and 4000 hours, with the environmental chambers cycling temperatures according to an externally programmed profile. This system was developed to automatically monitor and provide oversight to the various test chambers under the department’s control.
On an individual chamber basis, the system can verify that the chamber is performing to the test expectations, provide an audit mechanism and generate alarms when the chamber is not operating correctly. The software also is flexible enough to add and edit individual chambers and the tests inside them. The data collected is compared to set limits and, where appropriate, alarms are generated and events are logged to keep a history of what occurred during a test. The test system is capable of running many tests simultaneously.
The system is scalable and more thermal chambers can be added as needed. The operator can view the status of any given test by selecting the test to be viewed and observing the trend. The server software running on the server PC is tolerant of user logins and logoffs as it is running as an Windows service.
The software was written in LabVIEW as a client/server style application. Using LabVIEW and a small stub of “C” code, the server portion of the software was built into a Windows Service. There is no interface to the server other than the client. The client uses the LabVIEW VI Server technology to communicate with the server. This configuration allows the technicians to check the status of any test from their desk or a remote location.
Test configuration allows the operator to be notified when alarm conditions occur or for a regularly scheduled check of the chamber. The system notifies the operator by sending an email and/or by sending a message to their pager.
All test status information is persistent in an MS Access database so if a power failure occurs, or the system goes down, the tests in progress are not lost. When the system is powered up again, the system will restart any tests that were in progress. Two days of history data is kept in memory for each test so trends can be identified.
The system can generate a number of reports, such as job status, journal events, chamber status, completed test results, and chamber utilization. For each type of report, the technicians can pick from a list of criteria to filter the requested information.
At maximum throughput, the systems needed to consume during record and produce during playback about 800 MB/s/slot.
Client: A large company involved in C4ISR
A large company involved in C4ISR was developing a system for a new high-speed digital sensor device. Viewpoint was contracted to build a test system used in design validation and ultimately endurance testing of the sensor. Since the sensor was a component of a larger system which was being developed at the same time, another test system was created to simulate the sensor by feeding signals into the system.
Both the amount of data and the frequencies of the various digital signals were nearly at the limit of hardware capabilities. At maximum throughput, the systems needed to consume during record and produce during playback about 800 MB/s/slot. The FPGA clock on the FlexRIO had to run up to 300 MHz. The skew between triggers for data transmission needed to be less than 5 ns even between multiple FlexRIO cards even when the parallel data paths has inherent skews associated with the sensor. Finally, the systems needed to handle clocks that might be out-of-phase.
Achieving these requirements required significant engineering design in the face of multiple possible roadblocks, any one of which could have eliminated a successful outcome.
Furthermore, as usual, the development timeline was tight. In this case, it was a very tight 3 months.
To meet the timeline, we had to work in parallel across several fronts:
LabVIEW-based application development for both record and playback
LabVIEW FPGA development for marshalling data between the controller and DRAM
Custom FAM circuit board design and build
FlexRIO FPGA CLIP nodes and code for low-level data handling
This sensor had several parallel data paths of clock and data lines with clock speeds up to 300 MHz on each path requiring exacting design and build of a custom FlexRIO Adapter Module (FAM) and unique custom CLIP nodes for extending the FlexRIO FPGA capabilities. The FAM also had a special connector for interfacing to the customer’s hardware.
Additional NI hardware and software completed the system components.
The host application, written in LabVIEW, managed the configuration of the data acquisition and the control of the LabVIEW RT-based FlexRIO systems. The configuration primarily dealt with the number of sensor channels in use, skew settings between digital lines, and other parameters that dealt with the organization of the data passed between the sensor and the FlexRIO.
Two FlexRIO applications were written, one for record and one for playback. Each FlexRIO application was written in LabVIEW, and managed the configuration of the FlexRIO cards and the movement of data between the FlexRIO cards and the RAID drives. Note that Windows supported for the RAID driver. Between 10 and 32 DMA channels were used for streaming, depending on the number of sensor channels being used.
And, each FlexRIO application had an FPGA layer, written in LabVIEW FPGA enhanced with custom CLIP nodes. For the record application, we developed a custom DRAM FIFO on the FPGA to assist with the latencies on the PXIe bus. For the playback application, we were able to stream directly from DRAM.
The FlexRIO and stock FAMs from NI were initially considered as candidates for this project. Clearly, working with commercial-off-the-shelf (COTS) components would be most effective. Three options were available at the project start which could accommodate the required clock frequencies, but none offered both the required channel counts and skew/routing limitations. Hence, we had to design a custom FAM. This decision, made before the start of the project, turned out to be wise in hindsight because the parallel development path resulted in some shifts of sensor requirements which could be accommodated with the custom FAM but might have led to a dead-end with a COTS FAM.
In LabVIEW FPGA, a CLIP Node is a method to import custom FPGA IP (i.e., code) into a LabVIEW FPGA application. CLIP stands for Component-Level Intellectual Property. We needed to use special Socketed CLIP Nodes (i.e., VHDL that can access FPGA pins) for this project because we could expose additional features of the Xilinx Virtex-5 not exposed in LabVIEW FPGA by accessing Xilinx primitives. Some specific features were:
Faster FPGA clocking
Additional clocking options
Individual clock and skew control
Custom PLL de-jitter nodes
Essentially, the FPGA design had a majority of FPGA code developed in LabVIEW FPGA and we used CLIP Nodes for interfacing the signals between the FlexRIO and the FAM.
FlexRIO Adapter Module
As mentioned earlier, we had to create a custom FAM because of the need to route high speed signals from customer-specific high density connectors while synchronizing signals across multiple data channels and FPGA modules to within one (300 MHz) clock cycle.
At these high-speeds, the FAM needed careful buffering and impedance matching both on the signals as well internal components on the FAM PCB. At the start of the design, we utilized Mentor Graphics HyperLynx High Speed DDR signaling Simulation software to minimize signal reflections prior to building actual hardware. This step saved countless hours in spinning physical hardware designs.
We designed the FAM to allow channel routing and access to additional clock and trigger pins on the Xilinx chip and PXIe backplane.
The choice to base these digital record and playback systems on NI hardware and software was critical to completing this project. The open architecture in both hardware (custom FAM) and software (CLIP Nodes) enabled us to include some very creative extensions to the base toolset without which the project would not have succeeded in the allotted pressured schedule and on a predetermined budget. We were able to stretch the capabilities of the hardware and software very close to their maximum specifications by combining COTS and custom much more cost effectively than a purely custom design.
For this application, Dresser-Rand needed an extensible system capable of monitoring numerous signals interfaced to a large gas turbine. Well over a
thousand signals needed to be collected from an extremely varied set of data acquisition devices and instruments. The configuration of this system and
viewing of data needed to be available from any of a number of computers connected to the data acquisition network. Also, data needed to be available for additional processing on other connected networks. Dresser-Rand required that all of the components that were necessary to run a test, such as the server, database, acquisition, configuration, and viewing, were able to be run on one computer or distributed over several computers.
This system utilizes Client-Server architecture to acquire signals from a variety of devices and logs the data to a central SQL Server database. The data is then processed and viewed on remote terminals. It is modularly designed to facilitate changes in acquisition hardware as well as viewing and processing software. There are three important components to this application: a SQL Server data management system, TCP/IP packet based messages for configuration and data, and a flexible, applicationindependent driver model.
National Instrument’s LabVIEW was used for the bulk of this project. C, Visual Basic, and Fortran were also used to develop analysis routines and interface with various pieces of hardware.
TCP/IP packet based messages for communication of data and commands
100base-T local network with bridge to other company/worldwide networks
Remote configuration and viewing
SQL Server database
High channel count (1000+ signals)
Flexible data acquisition system
Diverse data acquisition devices: DAQ, GPIB, VXI, RS-232, PLC
Common driver model – drop in drivers, self-aware configuration
Common calculation model – drop in calculations, self-aware configuration
Flexible GUIs with drop in screens
Several software technologies used for various aspects of the project: LabVIEW, Microsoft SQL Server, Microsoft PowerStation Fortran, Microsoft Visual Basic, Microsoft C, Microsoft Access
Client: A major manufacturer of aircraft landing systems
A major manufacturer of aircraft landing equipment needed to develop a means of endurance and fatigue testing new designs for aircraft steering. The actuators involved in steering the nose landing gear (NLG) required precise and reliable control through thousands of steering cycles.
Control loops needed to be closed at faster than 1 ms.
Prior systems were handled manually without real-time control and monitoring.
Our customer designed and built a test rig to provide the hydraulics and environmental conditions for the endurance testing on the NLG. Viewpoint Systems supplied the electronic data acquisition and control hardware coupled with real-time software to provide the required fast control loops. The configuration and execution of the 1000s of steering cycles were managed by the same data acquisition and control system through a set of configuration screens that allowed specification of turn rates, min/max angles, drive and resistive torque settings, and so on.
The various PID control loop configurations were also configurable along with gain scheduling required under different operating conditions.
The environmental conditions were supported by controlling a temperature chamber through ramp and soak settings occurring during the steering tests.
Measurements on the steering performance were collected from commanded setpoints, sensor readings, and controller outputs during the entire test run.
Alarm and fault conditions, such as force exceedance, were monitored continuously during operation so that the system could safely run unattended.
The entire system underwent an extremely rigorous acceptance testing procedure to verify proper and safe operation.
Arbitrary Load and Position Profiles
Flight Position Control
Load Position/Force Control
Endurance/Flight Schedule Execution
Deterministic RT for DAQ and PID Control
PXI/SCXI Hybrid RT Chassis
Discrete Pump Skid Interface
Custom Control Panel/Console
Prior to deployment of our system, setup of a test was much more manual and operators needed to be around to monitor operation.
With our new system, complete endurance testing could be specified and executed with minimal supervision. Furthermore, the tight integration of real-time control and coordinated data collection made report creation much simpler than before.
The rigorous acceptance test gave trustworthiness to the data and allowed the design engineers to validate performance more quickly than the prior semi-automatic and manual methods of operation.
Setup of tests has been improved from prior operations. The endurance testing itself operated over a huge number of cycles lasting weeks to months between scheduled lubrication and maintenance.
The deployed system measures performance during the entire testing, even between the scheduled downtime.