CYGNSS Launch: The Human Side

If you were awake and/or scrolling CNN on December 15^th, you might have seen that a NASA mission loaded on a Pegasus rocket launched off of an Orbital ATK L-1011 Stargazer plane at around 8:30am Eastern time. Maybe you watched NASA TV as the rocket dropped away from the plane, hung in the air for five agonizing seconds, and then lit its solid fuel rocket and climbed up and out of view. You may have even watched long enough to see the deployment of all eight CYGNSS microsats from the deployment module take place on a simulated view animation. Even if you made it that far, it’s likely that you turned off the broadcast at that point and went about your day, convinced that the launch was successful and that the mission was ready to go.

It's a bit more complicated than that.

Inside the glass fishbowl of the mission operations center in Boulder, Colorado two teams of flight controllers, mission operation managers, mission planners, flight directors, system engineers, and subsystem experts worked day and night for nine days straight, ten if you include the day before the first launch window. As a flight controller on the mission I want to give you a rundown of what those ten days were like; the stress, the exhaustion, and the huge amount of work that the Launch and Early Operations (LEOps) team put in to make that the Cygnets made it into space just fine.

Mission Background

To me the Cyclone Global Navigation Satellite System (CYGNSS) is one of the more clever missions concepts around. Eight small satellites are equipped with delayed Doppler mappers, which receive reflected radio signals from Global Positioning Satellites. These are the same signals that you use to get GPS navigation and your satellite synchronous times. Observing the reflected radio signals for small changes in arrival time allows the CYGNSS scientists to calculate the height of waves that are completely obscured by clouds, which in turn is used as a measure of wind speed inside of a hurricane. The reflected signals are already there, CYGNSS just has the capability to measure them. By placing eight satellites just a bit larger than a kitchen microwave into orbit the mission can gather data from huge swaths of the Earth in hours, which will be especially critical during hurricane season. Before they can do all of these things, they need to get into orbit. And that is where my story begins.

December 11^th, 2016: One day to first launch window

After my last lazy morning for a few weeks and a late breakfast I came in to work and began updating all the computers to the final versions of our operations software and making last minute cheat sheets for everyone. There are handouts for all possible spacecraft identifications and all of the necessary phone numbers, there are lists of all possible points of contact and phones and head sets and radios for the official NASA launch communications. There are abbreviated versions of our contingencies for flight controllers to skim during passes and abbreviated versions of those abbreviated versions just in case there isn’t enough time to even read those. Essentially, if there was a procedure that could be condensed into an outline, and then again into a summary, both formats were printed out and available. I stayed until midnight checking over all of our systems with some other MOC ops personnel, even wiping down all of the screens in the MOC as well as the glass MOC door to make sure things were ideal.  Nervousness shows itself in interesting ways. Once the first of the night shift showed up at midnight I took the opportunity to go home for a few hours of sleep. Tomorrow was guaranteed to be a big day.

December 12^th, 2016: Launch Take 1

I set my alarm for a 5am wakeup so that I could be in the MOC by 5:30am, well in advance of the launch that was scheduled for 6:19am MST. A 5am wakeup after falling asleep at around 2am was welcomed only because I slept like a kid before Christmas, tossing and turning with excitement about the next day. The chatter of the NASA network about the status of the Pegasus launch vehicle and the Stargazer plane was broadcast to the MOC, and there was a projection of the NASA TV broadcast for visuals. The MOC doesn’t play much of a role in the launch itself since there is very little to do with the spacecraft until they reach orbit. However, we are quite invested in what happens after that point so the MOC main room had a display set up with some of the Pegasus diagnostics from Orbital ATK for our subsystem experts to keep an eye on. We sat there and listened to the chatter between the pilots and the ground control station as the plane continued on its racetrack flight pattern, watching as it passed the checkpoints for status updates. Over the comms from NASA we heard that there were issues with the hydraulic system that was necessary for launch. They restarted the system. Nothing. We heard a crewmember remark over the comm that he had banged on it a bit to loosen it up, with no result. CYGNSS was not going to space that day.

The MOC team immediately got together as soon as the launch was scrubbed and laid out a plan for the rest of the day. A launch delay for us didn’t mean taking the rest of the day off and coming back in for the next attempt, which was half expected by me. A launch delay was an opportunity to test, review, re-test, re-review over and over again. We heard that the next available launch window would be in two days, on December 14^th.

With two days ahead of us we ran more simulations and more tests, always on the lookout for the smallest issue that could cause problems with the spacecraft. Everything was tested again and again for consistency, making sure there was no special combination of variables that could cause errors or faults to trip. Around 6pm I decided to head out, knowing that I needed to catch up on sleep if I was going to be 100% for our launch on the 14^th.

December 13^th One Day to Launch, Take 2

There was something wrong. Gazes were downcast, the chatter of excited voices in anticipation of launch was diminished. I set my backpack and bags in my office and walked into the MOC. I got caught up on what had happened over night. During the testing we had been doing yesterday the flight software team caught a small but critical bug in the flight software parameters that had a decent chance of occurring. The December 14^th launch was postponed upon the discovery, and the flight software team got to work on the fix. Any update to the software needed the full gamut of tests run on it, so the MOC personnel were kept busy all day testing the updated software on the engineering model and software test bench for CYGNSS, making sure that these bugs were squished.

A review with Chris Ruf, the Principal Investigator, and John Scherrer, the Project Manager was scheduled for early on the morning of the 14^th. The CYGNSS flight directors and flight software team would need to convince them that CYGNSS would be ready to launch on the proposed Thursday launch window. If the PI and PM were persuaded that the risk had been removed, they would communicate this to NASA personnel just 30 minutes later and convince them of the same thing. Their work was cut out for them. The coffee pot was started for the umpteenth time that day, and the next shift came in to continue the testing where we left off. I headed home, exhausted and nervous of what a launch postponed into January would mean.

December 14^th One Day to Launch, Take 3

Getting up on this morning was difficult. This wasn’t so much from the fact that I was tired and wanted to stay in bed until noon, but the fact that I had been hyper-alert and prepped for launch for 3 days now. Being at that level of attention for 3 days straight and not knowing if the launch would be the next day or not was draining. I managed to put on some clothes and head to the MOC, just in time to catch the meeting that the PI, PM, flight directors, and flight software had scheduled with NASA officials.

The WebEx was heavily populated with anonymous call in users, all awaiting our evidence and conviction that CYGNSS was ready to go. I don’t know how much information I can directly convey from that meeting, so I’ll say the obvious.

They did it.

CYGNSS was a go. There was such a wealth of information provided on what the issue was, how it was fixed, and how it had been rigorously tested that it changed everyone’s mind. That’s right. Everyone. An entire conference call full of NASA officials changed their minds. Let that sink in.

The flight software and system engineering teams stayed all night, sleeping on a borrowed air mattress to make sure that everything was perfect. New simulations were run, initial conditions were tweaked, and in the course of 8 hours the software was completely impervious to this error or any error similar to it in nature. The software would be uploaded to the spacecraft that day and ready for launch the next day.

We were going to space again.

December 15^th Launch, Take 3

The thrill of launch overtook any exhaustion I complained about yesterday. In a déjà vu of Monday, the NASA TV was set up and live, the same deployment module telemetry was displayed, and the same cameras were displaying the same angles as before. We sat and watched as the L-1011 climbed to altitude and performed the same checks as before. There was a chuckle of relief when the hydraulic system gave the “Go” in the checklist and we were one step closer to launching. The L-1011 turned southeast onto the launch path and dropped the Pegasus, which was in free fall for five seconds before the engines ignited. The rocket sped off, slowly arcing up and out of view. We trained our eyes on the telemetry points from the deployment module that scrolled across our screen, looking for any hint of changes or problems. There were none, and we watched the successful deployment of all eight spacecraft off of the deployment module. Orbital ATK had done their job, and they had done it beautifully, placing our spacecraft into an orbit that was even a little higher than we were expecting. Armed with the vectors from ATK our orbital analysts and mission planners set about studying when our first passes could be. We settled into our chairs and started reviewing the pass plan for our contact that would happen in just under three hours.

So what do I, as a flight controller, do during these passes?

Prior to holding this job I thought that being a flight controller would mean holding joysticks and pressing lots of buttons. In reality, I look at plots and screens searching for values and type commands, or groups of commands called “procs” into a command line. These in turn are sent to a communication station and relayed to the spacecraft, with each spacecraft only accepting commands that are encoded for that spacecraft alone.

So in reality there’s only so much that can be done to prepare for a pass. There’s what you expect to happen, and there is what will happen. From our simulated early operations tests that we had done before I knew what procs would need to be queued up and we had contingency plans in place for a whole binder-full of scenarios. As long as the commands are ready to go and contingencies easy to access the only thing left to do is wait.

We had a predicted acquisition of signal (AOS) time that was provided to us by our orbit analysts using data provided to us by the Pegasus team at Orbital ATK. Within a few seconds of the predicted AOS the telemetry for our spacecraft started flowing and scrolling up the screen. A flurry of activity ensued as I communicated with the Mission Operations Manager (MOM) about the commands I was preparing to send, and the MOM notified the flight director who in turn was polling all the subsystems for their status. Without the headsets it would have been a shouting match. All the commands were sent and all the commands were received by the spacecraft, so our first pass was a complete success. Solar arrays were deployed, current was flowing, and CYGNSS was officially operational in low Earth orbit. There’s nothing quite like watching data scroll up the screen coming from a satellite a few hundred kilometers above the Earth, knowing that you are the first person to see it. That’s a cool feeling.

Throughout the rest of the day the CYGNSS team was able to make contact with all eight spacecraft, with seven of them being in ideal condition. One of them, however, had managed to get into an orientation after deploying from the Pegasus module that had the solar panels perfectly parallel to the Sun. This meant that the spacecraft was unable to generate much power and was working on battery power alone. After a quick meeting to discuss options to reorient the spacecraft a plan was put in place for our next pass that would hopefully get it out of the flat spin it had found itself in as the course sun sensor searched for the Sun.

By increasing the field of view for the course sun sensor with an uploaded command the team hoped that the onboard flight software would start to track the Sun. Even just the slightest change of power coming from the solar panels should allow the spacecraft to begin pulling itself out of the flat spin, so confidence was high that the fix would work. During our shift handover meeting the contingency procedure was reviewed and the night shift was briefed on the status of all of the spacecraft. I headed home around 8pm completely exhausted, barely able to eat before falling asleep.

The Days that Followed

For the next week two teams worked anywhere from 12 to 16 hour shifts for 24 hour coverage to start checking out each individual spacecraft. Power systems, attitude determination and control systems, the science instrument, all had to be tested again to verify everything was in order. The initial vector provided by the team at Orbital ATK to our orbital analysts was within a few meters of the expected, but continuous tracking by ground stations was necessary to produce updated two line orbital elements to help schedule future passes. Each day was some combination of these tasks, with passes coming every 20- 45 minutes.

The days blended together quickly, defined by activities rather than days of the week or hours. Time became relative, split between work and sleep. The one day that stands out to me was the day it snowed about six inches and I decided to skate ski to work, but other than that it was a steady pattern of up at 5:30am and in the office by 6am. I had a sleeping bag in my office ready for use, but at no point were things ever so dire that I felt the need to use it. As the days moved on the entire team became more efficient and machine-like, rattling off commands and procedures during passes like it was nothing. In between passes we would eat and work on updating procedures to eliminate the oddities that we had encountered in previous communication passes.

That brings me to one of the most entertaining aspects of the mission: food. Project managers from Southwest Research Institute had purchased enough food to stock a ship on an Antarctic voyage. An empty office was commandeered and designated the CYGNSS pantry, and was quickly packed with any artery-clogging, cloyingly sweet, or salty snack that you could imagine. It was glorious.

To keep the CYGNSS crew satiated during the week a white board in the office kitchen was labeled “CYGNSS Food Need List”. 

The night after launch, 4 days into our long shifts, the board looked like this:

Relatively innocent and representative of our state of tiredness and need for caffeine. About a week later the board looked like this:

At this point in the mission exhaustion was running wild. Our flight director reported to us that he had attempted to put his shirt on as pants and put two contacts in the same eye that morning. Coffee consumption was measured in gallons, cat naps in offices were common, and a funk had begun to cloud the MOC from the constant presence of humans for ten days straight. On Tuesday the 20^th we had our first successful automated passes with the spacecraft, proving that we were capable of downlinking data even if not physically present. The knowledge that the 24 hour operations were over and we could have our first days off gave everyone in the room a palpable sense of happiness. We Febreezed the living daylights out of the room for the first time in weeks knowing that the next shift would be there in 12 hours, allowing the fog time to clear. A skeleton crew monitored the spacecraft for the next few days, and something resembling normalcy was brought into our shifts. LEOps was complete.

When I first set out to write this blog I was hoping to capture the little human moments that always happen behind the scenes of operations like this. Growing up I had always known that behind every mission was a group of people sitting in a MOC making the wheels turn, but I had never heard anything about them. I wanted to tell that story. However, I realize now that the mission itself can get forgotten in this context. There is a very strong reason why people are invested in CYGNSS, one that will resonant with people from all backgrounds. At the end of the day, CYGNSS will save lives. The observations carried out by our eight Cygnets will provide the mission scientists with the best data to determine wind speeds with hurricanes, allowing better cyclone models and forecasting. That directly translates to better notifications for evacuations, earlier warnings, and ultimately saves lives. Despite the delays, the flat spin, and the exhaustion, CYGNSS made it to space healthy and happy. Earth science saves lives in the here and now as well as down the line, and I am proud to be able to say I helped get CYGNSS into orbit.