Thomas
Lövskog

Firmware Update over the Vacuum

CEO Additude Innovation

If Technology doesn’t fix the problem … we need more technology!


20th of August and 5th of September 1977 Voyager 1 and 2 was launched from Launch Complex 41 at the Cape Canaveral Air Force Station, aboard a Titan IIIE launch vehicle.

Their mission was to explore the outer solar system and they have brought back some amazing pictures of the same. Below is a picture of Neptune taken by Voyager 2 in the summer of 1989.

Recent Status

As of this writing 17th of October 2018, the voyager spacecrafts have been alive and kicking for 41 years ... FORTY-ONE-YEARS. They are whirling through space at a whooping 60 000 km/h or perhaps even more impressive 17 000 m/s. They have roughly covered 21 billion km since the launch from Earth.

In fact, they are so far from us that it takes 13 hours for our commands to travel to the spacecrafts and another 13 hours for a response to be received.

On August 25 2012, data from Voyager 1 indicated that it had become the first human-made object to enter interstellar space. Earlier this month NASA sent a note saying that Voyager 2 is now also about to enter interstellar space.

Considering this they are in pretty good shape. Both spacecrafts have 11 instruments each on board. Of these half has been turned off. They have energy for quite some time and are still sending data back to Earth. Around 2025-2030 they will run out of power.

Computers and Hardware

Although the spacecrafts was launched in 1977 it has some impressive specifications.

It has 3 computer systems. And two of each for redundancy. Today most of the redundancy is gone.

Computer Command System

The CCS is responsible for managing overall operations, including program sequencing, monitoring the health of the probes, communicating with the other computers, and uploading programs to memory. This is nearly identical to the computer used in the Viking spacecraft. 

The Viking CCS had two of everything: power supplies, processors, buffers, inputs, and outputs. Each element of the CCS was cross strapped which allowed for “single fault tolerance” redundancy so that if one part of one CCS failed, it could make use of the remaining operational one in the other.

The CCS could also operate in three modes: Individual (each CSS works on it's own task independent of the other); Parallel (both CCS work on the same task); Tandem (each CCS works on the same task independently).

CCS processors contained an instruction interpreter, data path control, and registers (18-bit accumulator, 12-bit link register, 12-bit program counter, 4-bit condition code register) which operated in serial. Instructions were 18-bit words, with the most significant 12-bits containing addresses (4k of direct addressing), and the least significant 6-bits containing operation codes (64 instructions). It had close to 70k of memory.

Flight Data System

The FDS is where engineering and science data was collected, formatted, and stored on the Voyager probes, and where all telemetry was collected. The development of the FDS began with a graduate paper which described the overall requirements of the FDS, as well as the trade-offs and benefits of hardware or software implementation, including which functions work best in each.

The FDS also handled the mass storage with 0.5 megabyte 8-track digital tape ... yes tape ... yes still working after FORTY-ONE-YEARS in Space. A freaking tape deck that has been storing data for 41 year in cold vacuum. Hats off for the engineers at NASA/JPL.

Attitude and Articulation Control System

The AACS of the Voyager probes determine the orientation, keeping it pointed towards Earth. The AACS has four routines that could be executed: scan platform stepper motors, thruster actuators, attitude control laws, and thruster logic. The AACS also sent a “heartbeat” to the CCS at regular intervals.

Broadband in Space ... not so much

Well there is no gigabit in space. However, you may still be impressed that the playback of data and transmission from the internal 8 track digital tape storage was between 7.2 kbps to 115.2 kbps. In 1979 when the spacecraft passed Jupiter it could transmit with 115.2kbps, in 1981 when Voyager 2 passed Saturn the transmission was at 44.8 kbps.

Nowadays the maximum data rate is 7200 bps and in the future it will be limited to 1400 bps.

Command and telemetry are transmitted and received at 160 bps and 40 bps. 

Firmware Update Extravaganza 

As stated above FDS like the CCS is programmable in-flight. That is 1977 words for FOTA (Firmware upgrade Over The Air). I guess FOTA couldn't be used, because the lack of air ...

Usually a small PIC, Cortex-M0/3/4, Renesas RL78 or similar have roughly the same flash sizes as Voyager. This is non-volatile memory. When you do firmware update you first typically download the code to internal or external flash, reboot the MCU, it finds the new flash, verifies it and burns it to the internal MCU. Then verifies the internal flash and if ok, executes. Or (and many MCU has on-board support for this) you split the flash memory in two and if the current active flash is in the lower half, any firmware updates are written to the upper. Once verified the MCU switches the active flash area including all reset, interrupt and similar vectors.

If you brick it, you need to open your electronics and flash using JTAG or similar approach. Costly, annoying, but doable.

This is 2018 still a very delicate and complicated process. And wasting half the memory is a bit naughty.

41 years ago, it probably was a bit frightening. Especially considering ...

  • Bricking is simply not an option. Really not an option.
  • Your space craft is billions of km away, moving extremely quickly.
  • You are limited to 160 bps up-link for data, and it takes 26 hours from transmission to reception of "OK!"
  • Voyager also used volatile CMOS memory. So if they loose power, they loose the program.

Still they managed to do some remarkable updates. Here are a few ...

New tools, new languages

The original spacecraft software was written in Fortran 5, then it was ported to Fortran 77. Now that is fear. To actually port software, compile it, and send it away waiting days to see if it was received and executed correctly. Brutal. Then after a few years they did the same feat again. Some of the software is now ported to C. One of the reasons is the need for more compact software ... which takes less time to send over the link.

Impressive feat 1!

Passing the outer planets - wanting better images

As the voyager was passing through the solar system the usable bandwidth fell. At Jupiter pictures could be downloaded with 115.2 kbps, at Saturn they where down to less than half of that. The camera still takes the same high-res pictures and you want more.

The engineers again used the software in-flight update. With a better better compression routine they could store more with less. Voila! More pictures despite lower bandwidth.

Impressive feat 2!

Passing the outer planets with a broken camera mount

When passing the planets furthest from the sun, the mission team realized that the camera mount was damaged. They could not position the camera for the perfect picture. What do you do ... what do you do.

The engineers knew they had a bit more fuel than anticipated, so instead of turning the camera they rewrote the software to spin the actual spacecraft instead. With the risk of spinning it out of control and the antenna turned from Earth that REALLY had to have been a long wait until they realized everything went ok.

Impressive feat 3!

The flipped Bit!

On April 22 2010 the mission team members noticed that data packets from Voyager 2 was changed and not what it was supposed to be.

On May 6 engineers shifted the spacecraft into a mode that transmits only spacecraft health and status data while they diagnose an unexpected change in the pattern of returning data. Preliminary engineering data received on May 1 show the spacecraft is basically healthy, and that the source of the issue is the flight data system, which is responsible for formatting the data to send back to Earth. The change in the data return pattern has prevented mission managers from decoding science data.

On May 12 the commanded Voyager 2 to dump the entire memory, billions of km from Earth, to be able to compare it with what it should be. Something was corrupt.

The engineers isolated the one bit in the memory that had changed, and they recreated the effect on a computer at JPL. They found the effect agrees with data coming down from the spacecraft.

May 19 they managed to send a reset command to Voyager to flip the failed bit. After monitoring it for a couple of days, they turned on all the scientific instruments on May 23. 

Impressive save and feat 4!

Conclusion

I will never complain over FOTA work for a consumer electronics device on Earth again. I will happily design, write and debug.