What Caused the Proton Rocket to Crash?

[nsaspook]

http://www.russianspaceweb.com/proton_glonass49.html#culprit

By July 9, it is transpired that investigators sifting through the wreckage of the doomed rocket had found critical angular velocity sensors, DUS, installed upside down. Each of those sensors had an arrow that was suppose to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead. As a result, the flight control system was receiving wrong information about the position of the rocket and tried to "correct" it, causing the vehicle to swing wildly and, ultimately, crash.

Seems like a simple mechanical interlock (notch or flat) could have prevented this problem. http://xkcd.com/1133/

 

[D H]

Each of those sensors had an arrow that was suppose to point toward the top of the vehicle, however multiple sensors on the failed rocket were pointing downward instead.

That is very close to Murphy's Law in its original form: "If an aircraft part can be installed incorrectly, someone will install it that way."

Murphy's Law has mutated since then to "If anything can go wrong, it will." But that's just Murphy's Law operating on Murphy's Law.

 

[SteamKing]

Must've been the new guy's job to install these parts. This wasn't the first Proton booster built or flown. Maybe the QA guy was upside down when he signed off on the assembly.

 

[etudiant]

It is actually a real testimony to the skill and dedication of the workers in Russia's space industry that these kind of errors are not more frequent. Building a quality product in facilities starved for investments while getting paid badly or not at all is not easy. The miracle is that most of the boosters still work ok despite these impediments.

 

[nsaspook]

It is actually a real testimony to the skill and dedication of the workers in Russia's space industry that these kind of errors are not more frequent. Building a quality product in facilities starved for investments while getting paid badly or not at all is not easy. The miracle is that most of the boosters still work ok despite these impediments.

That might be true but installing the critical sensors upside down without that being detected required true FUBAR genius and inattention to detail not often seen today.

http://www.youtube.com/watch?NR=1&feature=endscreen&v=YuiPvGQapcY

 

[riscoe]

I can give you a technician's take on these scenarios-
I am an actively employed corporate jet aircraft mechanic (also worked on commuter airline aircraft).
Certainly there is a current shortage of experienced technicians in the worldwide market- the emerging Asian aviation industry has been drawing techs from all over the globe, and I would guess especially from nearby Russia. But we can't blame everything on the "new guy".
I've observed that the best techs are those that really enjoy what they're doing, but this comprises only a small percentage of any given sample of employees. Most are there for general career reasons, knowing tech jobs pay better, or offer better opportunities into other careers or directions. The remaining employees are those that should have stayed as non-skilled labor, but got through for various reasons.
The industry tries to accommodate the shortcomings of employees by adding personnel solely for inspection, adding detailed written procedures for each task, and even idiot proofing parts and assemblies. (The last one gives rise to entities hiring idiots who find ways around the first 2 ideas- I quit being surprised by this.)
Even the best techs make mistakes-
Sometimes one can be so focused on one element of the task, and miss something obvious as an arrow on a component.
Something as simple as an interruption of the tech during a task can cause mistakes.
I think the following personal experience is likely similar to the cause:
I was reinstalling fire extinguishing components for an auxiliary power unit and received an ok to install from a very good lead mechanic with inspection privileges (one of the guys that loves aviation). For some unknown reason, I took a closer look at the same items the lead had just examined, and found a large (2 inch long, 1/8 inch wide) crack in the extinguisher tube. I couldn't believe the lead missed it, so I called him back over and asked him to look at everything again and see if he still wanted me to continue installation. He looked at everything and said to proceed. I pointed out the crack and he was shocked that he missed such an obvious defect. He subsequently called over one of our better inspectors and asked him to give the ok to reinstall the extinguisher components. The inspector also missed the very same crack, and was just as shocked when we pointed it out. I can only conclude that the defect was missed by 2 exceptional techs because they didn't EXPECT to find it. I didn't expect to find it either, but some hunch told me something was wrong. Hunches are funny things. Nobody knows how to make them happen.

 

[SteamKing]

What's Russian for 'This end up'?