Thomas
Lövskog

Reboot saves the day ... or?

CEO Additude Innovation

If Technology doesn’t fix the problem … we need more technology!


We have all gotten the following support answer to a IT problem ...

- Please reboot the system.

As an engineer that is a major let-down for two reasons ... 1) That absolutely gives me no information of what was wrong in the first place and 2) When it works, and it does sometimes, it is living proof for poor engineering work.

Now this surely is no tactics used and needed by NASA ... but maybe even the sun has spots? This is at least the story recently in the tabloids here in Sweden. As always the finer details revealed a little bit different story than "just reboot" ...

... and as always in the process of research, I stumbled into a world of some, seriously cool engineering.

 

The Hubble Space Telescope (HST) is a space telescope that was launched into low Earth orbit in 1990 and remains in operation. The telescope is named after the astronomer Edwin Hubble and is one of NASA's Great Observatories.

Below is 2 of the famous Hubble pictures. V838 Monocerotis and Supernova remnant N49. Here you can watch 27 of the best pictures taken by the Hubble telescope.

Hubble and positioning

Obviously positioning is crucial for a telescope in space with very long exposure times, faint light, distant objects, and everything that makes your work really really tricky. It takes a bit of effort to view billions of light-years away.

The Hubble positioning system is also one of a kind and unique for, again, obvious reasons.

Sensors

Hubble have 5 different sensors to determine and measure how it moves. And they all serve a very specific task. 

  • Coarse Sun Sensors
  • Magnetic Sensing System
  • Gyroscopes
  • Fixed Head Star Trackers
  • Fine Guidance Sensors.

The Coarse Sun Sensors determine the orientation of Hubble in relation to the Sun. It consist of silicon that measures light to determine where the sun is located. Hubble was created to see very faint light and the sensitive equipment must be pointed away from the direct sunlight at all times. Even the lining inside the telescope can be damaged by sunlight.

The Magnetic Sensing System acts as Hubble’s compass. It measures the telescope’s orientation in relation to Earth’s magnetic field.

The Gyroscopes, the heart of the story here, measures Hubble's angular movements. They are top notch and the best of the best gyroscopes ever designed.

Mechanical gyroscopes are based on a principle discovered in the 19th century by Jean-Bernard-Léon Foucault, a French physicist who gave the name gyroscope. The gyroscope is a wheel suspended in gimbal rings. The angular momentum of the spinning wheel caused it to maintain its attitude even when the gimbal assembly was tilted.

Hubble's gyroscopes is constructed as a wheel that spins at a constant rate of 19,200 rpm on gas bearings. This wheel is mounted in a sealed cylinder, which floats in a thick fluid. Electricity is carried to the motor by hair-thin wires that are also immersed in the fluid. Electronics within the gyro detect very small movements of the axis of the wheel and communicate this information to Hubble’s computer.

The Gyroscopes in Hubble is also mounted in pairs that is called a Rate Sensor Units (RSU). Hubble has 3 of these RSUs for a total of 6 gyroscopes. Hubble needs 3 gyroscopes to work efficiently, but it can handle it's operation with 2 and even 1 with limited performance.

The gyroscopes have been replaced 3 times. In 1993 on the first servicing mission two RSU (4 gyroscopes) were replaced. In 1999 all units were changed. Finally in 2009 all RSU again were changed and upgraded.

The Fixed Head Star Trackers determine Hubble’s attitude by measuring the locations and brightness of stars. The pattern of stars are matched by the on-board computer to position and point Hubble correctly.

The last sensor is The Fine Guidance System. This is Hubble’s most accurate positioning sensor. The FGSs (3 of them are on-board Hubble) use starlight captured by the telescope’s to maintain a fix and lock on guide stars. This ensures that the spacecraft’s attitude does not change. The fine guidance sensor is so accurate that it is the equivalent of maintaining a laser-pointer on a small coin 320 km away.

Actuators

Now we know how absolutely magnificent Hubble's sensors are. Now the question is how do you re-position the actual telescope? Rocket thrusters perhaps ... nope! The propellant and exhaust would compromise the telescopes image. So what to do?

In space it is technically difficult in one way to maneuver,. On the other hand other means are possible when friction, gravity and air is not present.

Hubble uses two actuator systems. One super cool and one insanely cool.

I am pretty sure that my admiration for the engineering work shines through, but the second here is .. well ... I do wonder how the innovation process of coming up with this stuff are.

Newton’s third law of motion states that for every action, there is an equal and opposite reaction. An astronaut throwing a wrench in front of himself will be pushed in the other direction as well. Hubble utilizes this principle with four massive wheels called reaction wheels. These are the size of your ordinary tires but with high mass. If one of the reaction wheels turns clockwise, Hubble will turn counterclockwise, and vice verse.

Despite their relatively small size, they can easily turn Hubble with relative ease. About 90 degrees around one axle in 15 minutes.

Now to the really cool way of moving.

Hubble have four Magnetic torquers installed at 90-degree intervals on the outside. These are eight-foot iron cores wrapped in wire that produce a magnetic field. This magnetic field can push or pull Hubble relative Earth’s magnetic field, rotating the telescope. How ingenious isn't that?! Your ordinary play with magnets but on a whole different scale.

With these 5 sensors and 2 actuators Hubble can seek out and focus on objects with unrivaled accuracy. Allowing it to observe objects more than 13 billion light-years away.

The Problem

On October 5th, 2018, the Hubble Space Telescope entered safe mode, owing to the fact that one of the three gyroscopes used failed. The gyroscope that failed had been showing signs of trouble for about a year, so that was not a real surprise.

Before this happened Hubble had 4 Gyroscopes known to be functional. The one spare was actually turned off many years ago, because of behavior that was not up to specifications. Now the engineers tried to get the spare up and running, but it was still not behaving as expected.

The problem was that the gyroscope reported rotation rates that are orders of magnitude higher than they actually are. Tests done showed that the gyroscope properly tracks Hubble’s movement, but the rates reported are consistently higher than the actual rates. This is similar to having your cars speedometer showing a constant 200 km/h higher than reality. Relative speed changes can be tracked, but the absolute speed is way wrong.

Worth to note is that this spare had now been turned of for 7.5 years and left silent in the vacuum and cold of space.

ReBoot ...

One of the thoughts that dawned on the engineers was that perhaps the spare gyroscope hadn't started up from deep sleep as it should. The engineers therefore tried to reboot the gyroscope on the 16th of October. They did this by switching it off briefly for a second, and then restart it before the wheel spun down. However, the resulting data showed no improvement in the gyroscope's performance. 

Another thing that could be a potential problem was blockage/clogging in the fluid surrounding the gyroscope. On October 18, the Hubble operations team commanded a series of spacecraft maneuvers. They tried to "wiggle" Hubble to attempt to clear any blockage that may have caused the float to be off-center. The term "wiggle" might be a bit off for a space-telescope the size of a bus. 

This actually seamed to help and they could see differences in the behavior .. to the better. So on October 19th they repeated the maneuvers and it seamed to totally clear the gyroscope from any blockage.

The Hubble operations team plans to execute a series of tests to evaluate the performance, including moving to targets, locking on to a target, and performing precision pointing.  After this Hubble is expected to return to normal science operations.

So ... a re-boot was involved, but it was a bit more than that. NASA engineering image preserved ;-)