FIRST IN A SERIES
When SpaceX debuted the Block 5 version of its Falcon 9 rocket in 2018, it expected to fly the reusable boosters 10 times before taking them out of service for major refurbishment. But last summer, the company quietly moved the goalpost.
“We got to 10 [flights] and the vehicles were still looking really good, so we started the effort to qualify for 15,” Jon Edwards, SpaceX vice president of Falcon launch vehicles and Falcon engineering, told Aviation Week during a series of interviews with top company managers.
Meanwhile, the time between Falcon 9 launches continues to shrink, with SpaceX now aiming to fly from one of its three launch sites at least once every five days. On April 29, the company launched a booster that had landed just 21 days earlier, besting its previous turnaround record by six days. This month, the Falcon fleet leader will attempt a 13th flight.
- SpaceX makes headway in maintenance, repair and overhaul
- Starlink launches serve as flight testbeds
- Rockets now certified for 15 flights
The space launch industry is still a long way from airline-like flight operations, but momentum is growing. SpaceX, established in 2002 to fulfill founder, CEO and chief engineer Elon Musk’s heady plan to colonize Mars, is not the only space company building reusable hardware, but for now it continues to be the industry juggernaut.
In this two-part series, Aviation Week shares an exclusive look at how SpaceX is turning the long-held wish for operational reusability into reality and what other space companies, including the Boeing and Lockheed Martin-owned United Launch Alliance, are planning.
The SpaceX Way
This month, SpaceX plans to launch a Falcon 9 rocket for an unprecedented 13th time. If it lands successfully, extending an unbroken string of 49 successful touchdowns as of June 8—and 123 successful landings overall—the fleet leader could fly two more times before year-end.
The operational 21-member fleet—consisting of 10 single-stick Falcon 9s, two converted Falcon Heavy (FH) side boosters now flying as single sticks, five FH side boosters and four FH center cores—includes two other 12-flight boosters and another that has flown 10 times. “We don’t need a huge fleet in order to do the launches that we’re doing,” says Richard Morris, SpaceX vice president of production and launch.
“Our goal is to be able to launch at a five-day rate or faster,” Morris says. “To do that, we look at every piece of the process flow—booster maintenance, the launchpad, recovery team, fairings—and look how to optimize and speed that up. We have this approach where we iteratively improve each process and shrink the time from launch to launch. That’s how we’ve been able to do this.”
SpaceX currently plans to retire the boosters after 15 flights, but that could change. The company has committed to supplying Falcon 9 launch services to NASA and the U.S. Defense Department through at least 2030. SpaceX intends to transition its commercial customers, along with its own Starlink satellite launch business, to the fully reusable, super-heavy-lift Starship system currently in development.
“In the very beginning, we thought the [Falcon 9 Block 5] rockets might be good for dozens of flights, but we needed to pick something to qualify it to,” Edwards says. “We wanted something ambitious that gave us a reasonable target, so that’s where the 10 flights came from.”
As the fleet leaders hit their ninth and 10th flights in 2021, SpaceX began requalifying Falcon 9 components for 15 flights. “We did these delta qualifications before we started on those additional flights so we know we’re putting our best foot forward and that we’re not taking on too much risk as we’re pushing the envelope,” Edwards says.
The testing entailed putting every flight component, including batteries, flight computer, avionics boxes and propulsion components, onto a shaker table for vibration testing to four times the fatigue life of what the hardware would experience over 15 flights. “That gives us margin well beyond what they’ll ever see in their service life,” says Jessica Jensen, vice president of customer operations and integration.
“We’re not going into a 12th or 13th flight crossing our fingers,” she adds. “We have confidence based on all the testing we’ve done on every piece of hardware that it’s going to survive.”
The decision to requalify the boosters for 15 flights was a natural outgrowth of the ongoing evolution of the Block 5. “Every flight, we’re continuously inspecting, learning and then reapplying those lessons to either changing the design, a manufacturing process or our inspection methods across the fleet and into the next flights,” says Kiko Dontchev, SpaceX vice president for launch.
Analysis identified a few components that will need to be studied on a case-by-case basis as the vehicles surpass 10 flights, notes Bill Gerstenmaier, SpaceX vice president for build and flight reliability. “As we start getting into the 12th and 13th flights, we’ll take a more invasive look to see if there is any life degradation in those areas,” says Gerstenmaier, who joined SpaceX in February 2020 after a 42-year career at NASA.
“So far, we’ve been pleasantly surprised we’re not seeing a lot of degradation in those areas,” he adds. “So the analysis was conservative; but we’re watching.”
Proactively, SpaceX has been replacing some components, including the booster’s water-cooled engine boots, which are flexible parts of the aft thermal shield that protects engine components inside the engine bay from the re-entry environment.
Another area of interest is a movable transfer tube that funnels oxidizer down to the engines. “It moves up and down, and it looks like there’s some wear there,” Gerstenmaier says. “There are three seals there, and we can actually pick up a leak between the seals. We may have to change out some of those seals at some point, but we’re monitoring all that stuff.
“What we’re more cautious about is the unknown unknown—something where the analysis says that we have a lot of margin, but the operational environment is a little bit different than what we have in our models and we don’t know that,” he adds. “We continually keep looking.”
SpaceX has three classes of inspections: A class is performed for every mission; B class involves periodic maintenance, which is now performed every sixth or seventh flight; and C class, the most thorough maintenance process, is used for the fleet life-leaders and for all crewed missions.
“A lot of the work we’re doing is preventative stuff, like inspections,” Edwards says. “In theory, you could bypass those once we’ve learned everything we can learn, but as of now we don’t really have to go there. We have the time to do these inspections, so we don’t want to descope those quite yet.”
Gerstenmaier is working to shift the reliability focus from inspections at the end of the process to how the vehicles are manufactured and assembled. “We’re trying to change the beginning of the process . . . so that a high-reliability product comes and then you don’t have to worry about doing the inspection at the end,” he says.
“I can remove inspections and I’m not increasing the risk because I have a much more repeatable, reliable process that actually puts the hardware together,” he adds.
Dropping requirements that have outlived their usefulness is an important part of SpaceX’s improvement process. “We’re kind of in a continual learning mode,” Gerstenmaier says. “We let the hardware talk to us and tell us what’s going on, what’s changing and where it makes sense to remove stuff because it’s just taking time and that’s the right thing to do.”
For example, the company changed its requirement to perform a static firing every time an engine was removed.
“Now we have to drop three engines before we go do a static fire because we determined we can put the high-pressure joints together, verify that they’re leak-tight and everything works,” Gerstenmaier says. “We’ve removed like 14 static fires out of the flow that would have been required under our old criteria, and all 14 of those flights launched on the first attempt.”
SpaceX also removed requirements to perform static engine firings after a turbine wheel is changed out. Instead, the company added some accelerometers on the engine to verify that the turbine wheel is operating properly. In case of any problem, the instrument can trigger an automated abort up to the last half-second before launch. “We use the actual launch countdown to effectively replace a portion of the static firing,” Gerstenmaier says.
The inability to drop requirements was a primary reason why NASA’s space shuttles were never able to come close to projected flight rates, notes Gerstenamier, a former NASA associate administrator who oversaw the space shuttle, International Space Station and other human space-flight programs.
During the space shuttle program’s peak years of 1992-97, NASA flew seven or eight missions per calendar year. In 1984, it flew nine times.
“We got forced to be conservative for a variety of reasons, and we could never remove requirements,” Gerstenmaier says. “I attempted to remove requirements, and I was unable to, or it would take me 10 years.”
For example, early in the shuttle program, NASA needed to pull the shuttle main engines after every flight for inspection. But after several flights, the inspections were not revealing any issues. “They weren’t adding any value, and I wanted to stop the inspections,” Gerstenmaier says. “But we had gotten so good at pulling engines, the program said, ‘Why don’t we just pull engines and go look because we can?’
“At the end, we were tearing apart all these shuttle engines for inspection and we ended up operating at the low end of the reliability curve,” he says. “We actually wore out components during testing and put more life on them than we did in actual flight.
“If you’ve got hardware that is ready to go fly, you’re better off not tearing it apart to inspect. To understand if it has a problem, you use the reliability of the hardware to drive you and you only inspect when you start getting out to that later life period,” he adds. “We weren’t allowed to do that in shuttle.
“At SpaceX, we see the benefit of removing a requirement, and then if we remove too much, we can add it back in—no harm, no foul,” Gerstenmaier notes.
“We have the agility to put stuff in that makes sense and adds high value,” Gerstenmaier says. “If it doesn’t make sense, we can pull it back out. It’s a very small engineering team with very low overhead that can respond in the appropriate way to what the hardware is telling us to do.”
NASA used to operate in a similar fashion, but as the agency matured and encountered problems, it became more conservative. “Other folks looked at us, and we got more bureaucratic,” Gerstenmaier says. “It’s a natural thing at a company, but I want to make sure that doesn’t happen at SpaceX. We need to keep that same agility, that same drive for engineering excellence.
“It’s not easy, because you get in this mode where you say: ‘I’ve always done it this way; why would we ever want to change?’ But you’ve got to stay hungry—or as SpaceX says, stay paranoid—and keep looking for what you can do better,” he says.
For example, the Block 5 launcher, unveiled in 2018 as the Falcon 9’s final version, has continued to evolve. Hardware changes include an upgraded aft thermal shield and a modification to the upper stage so it can dump propellant if an umbilical separates from the quick disconnect (QD) plate. “It’s hard to imagine how the QD plate might accidentally pop off, but we knew that the result would be catastrophic, so we implemented the change,” Edwards says. “Most changes at this point are intended to improve system reliability and/or rapid reusability of the vehicle.”
SpaceX continues to make operational changes as well, thanks largely to data collected during Starlink deployment missions. In addition to spurring SpaceX to dramatically increase its flight rate, Starlink launches provide a relatively low-risk testbed to better understand the Falcon’s flight envelope.
“If there is something we don’t fully understand, we will test it off-line to understand the risk, fly it on a Starlink flight to gain real test data and then apply that learning to our other customers,” Gerstenmaier says. “Pushing the envelope from a performance standpoint actually makes us more reliable for crew and other missions because we’ve been able to kind of experiment with our hardware a little bit.”
With the June 8 launch of Egypt’s Nilesat 301 geostationary communications satellite, SpaceX has racked up 157 Falcon 9 flights, 99 of which launched on reused rockets. One-third of the Falcon 9 missions were devoted to deploying SpaceX’s Starlink satellite network, which is designed to provide high-speed, low-latency internet connectivity worldwide. “The Starlink missions are our highest-heating missions, and we’re flying our life-leaders on them, so that’s where we’re learning the most,” Edwards says.
SpaceX is using the Starlink missions to modify the Falcon 9 thrust profiles, providing more insight into vehicle reliability and operating margins. Engineers also applied lessons learned from two premature engine shutdowns during Starlink launches in March 2020 and February 2021. In both cases, the remaining eight Merlin 1D engines compensated for the malfunctions, but the boosters were lost during landing attempts.
In the first incident, which occurred on March 18, 2020, engine No. 3 shut down 140 sec. after launch when residual fuel—most likely cleaning fluid from a gas generator flush refurbishment procedure—ignited in a high-pressure oxygen section of the generator, causing it to fail. It was the first failure of a Merlin 1D engine, which debuted in 2013.
An earlier Merlin variant, the 1C, failed during SpaceX’s fourth Falcon 9 flight in October 2012, which was the company’s first cargo resupply run for NASA. That booster also successfully delivered its primary payload, a Cargo Dragon capsule, into orbit.
Another Merlin 1D, this time engine No. 9, which sits at the center of the Falcon 9’s “octaweb” design, shut down 143 sec. after launch on Feb. 16, 2021, when hot gas breached the engine bay, causing an electrical short in the throttle valve harness. That led the gas generator to fail.
“We determined that the plume recirculates and can actually come back around engine 9,” Gerstenmaier says. “A hot jet essentially got into that engine compartment, burned through a harness and shorted out some wires. Then the valve went full open and we had an engine shutdown.
“We were close enough to orbit and were able to recover and it worked out, but we clearly had an engine failure that was caused by this harness problem,” he adds.
SpaceX resolved the issue by shielding the harnesses and changing its fault-detection software, enabling the rocket to return to flight. Engineers also began working on a long-term solution to prevent jets from reaching that location.
“This is an example of where we went beyond just fixing the problem,” Gerstenmaier says. “We not only shielded that harness, but we also shielded a bunch of other harnesses that had potential susceptibility. Then we found this software problem that didn’t protect for off-scale low [voltage] on these sensors, and the software reacted to that in the wrong way. So we fixed all that software as well. We made other system changes that were unrelated to where the failure occurred.
“If the vehicle gives you a problem, it’s an opportunity to use imagination and . . . prevent a future failure you didn’t even know was coming,” he adds. “You not only fix your first problem, you fix two other problems that haven’t even occurred yet.”
SpaceX also has used Starlink missions to experiment with the timing between first-stage separation and second-stage engine ignition. Separating the first stage and immediately lighting the second is better from a second-stage performance perspective, but the plume then hits the first-stage interstage components, increasing wear. “We’re looking at how you light the engine the earliest, but not damage components in the first stage,” Jensen says.
So far, SpaceX has worked down to a 1,000-in. separation by decreasing the time between first-stage separation and second-stage ignition in increments of 0.25 sec.
Similarly, Starlink missions provide SpaceX a testbed for fairing separation timing. Customers typically have strict requirements on when the fairing is deployed to prevent rarified atmospheric particles in the fringe of space from heating the payload.
“On Starlink missions, we incrementally started deploying the fairing earlier and allowing higher and higher heating,” Edwards says. “Now we’re at the point where it’s more than 10 times the heating that is typically allowed on an external customer mission.”
One System To Guide Them All
SpaceX’s ability to test, analyze and build so quickly rests on an electronic test and processing system named Warp. To build a rocket, technicians input data into a computer, a process that Gerstenmaier can monitor. “If I see something I don’t like, I can go down to the floor and talk to them,” he says. “That’s unheard of in my past life. I would get some paper product like a month later, and I could go look through it and see where somebody didn’t write down the right torque value.”
The data system includes an automated data-checking process called Hyperion that takes data coming back from a rocket to make an initial cursory check to see if all the parameters are correct. “It errs on the side of calling something a failure,” Gerstenmaier says. “That then forces the engineer to go and look and understand why a particular measurement is out of spec from the previous flights. It removes the burden from the engineer of looking at all the data by essentially pulling the data that’s off-nominal.”
Another electronic tool handles workflow, which in effect lets SpaceX know when it is ready for flight. “In my past life, that was an all-paper process with a ton of meetings,” Gerstenmaier says. “Now it’s all electronic data you can go parse through and figure out where you are. You can see trends, you can pull data. We have a tremendous amount of metrics that help us now look at reliability with a different light.”
With SpaceX’s all-encompassing Warp system, if an engineer needs to make a change, he or she would open an issue ticket and document, for example, that post-flight inspections found a particular issue with a particular part.
Engineers would review that and Warp would flag similar issues that may have occurred on previous flights. To make a change, the engineer opens a change ticket, which would then be reviewed by several people to make sure the modification will not impact some other system.
Warp then automatically kicks off the planning process by listing the new parts to be built and issuing work order instructions. “It’s all in one huge enterprise system that SpaceX built,” Jensen says.
“This is the advantage of a company starting out from the beginning and also having a strong software background,” Gerstenmaier adds. “We also benefit tremendously by looking at what the automobile industry does and how they achieve high reliability. We’ve hired some folks on the production side from the automobile industry, and we put that expertise in place.”
For Starlink, SpaceX is tapping best practices from the commercial electronics industry. “We’re building 20,000 home-user Ku-band phased array antennas out here for the Starlink system per week. That’s unheard of,” Gerstenmaier says.
The lessons learned get folded back into engine production as SpaceX gears up to build Raptor engines for Starship vehicles at a rate of one every 18-24 hr. “We’re able to essentially break down barriers that used to be typical stovepipes,” Gerstenmaier says. “We integrate, take best practices, innovate and stay agile.”
Related Content
Comments
Thank you.