How Safe is the Boeing 737 Max's MCAS System?

[anorlunda]

Sigh, long threads are tedious because things posted earlier may be forgotten.

This all makes it hard for me to imagine the lack of redundancy in the Max MCAS system

Earlier in this thread this was discussed. The Max had fully redundant A and B strings, with manual switching between them. In the accident planes, if the pilots had switched from A to B, the crashes may have been avoided.

The complaint is that A string has one AOA sensor, and B has one AOA sensor, but some people complain that both strings should have access to both sensors without manual switching. In that sense, A and B strings would no longer be fully independent. Cross-connections between strings introduce mutual dependencies and new kinds of common mode failures.

Boeing's design can be criticized, but it is unfair to characterize is as lack of redundancy.

If they'd done things right from the start, the 737 MAX would probably not have existed

Or, alternatively, if Boeing had been willing to spend the amount of money it would have taken to do an MCAS-type system right

The still better alternative, was also mentioned upthread. If I remember right, it was the option to redesign the landing gear to allow higher ground clearance instead of moving the engines forward. It was mentioned upthread that the new landing gear design had been completed for the 737-MAX-10 but not used in 737-MAX-9, thus leading to MCAS.

 

[russ_watters]

The inevitable result of these ongoing tradeoffs is to eventually produce an engineering product that is too labyrinthine to analyze.

I think that's a stretch, and really weird to apply a such a harsh value judgement such as "reprehensible" to it. How would you even measure such a thing? Ultimately the 737 Max is back in service and it still has MCAS. I don't know if the 737 Max is more or less complex than an A380 or 787, but I suspect it is substantially less complex. It should have been possible to make it work the first time.

 

[hutchphd]

I don't really understand your argument. Yes they have cobbled together a fix for the Frankenstein and it will rise. But 346 people are dead. My argument is that the proximate cause should not be substituted for the fundamental cause. The proximate cause was bad design verification and validation; the fundamental cause was choosing profit over good design practice. The proximate cause mistake is regrettable but the fundamental cause mistake is reprehensible.

 

[russ_watters]

I don't really understand your argument.

It's more confusion than an argument. I don't understand applying any value judgement at all to the idea of complexity. Systems can be simple or complex or really really complex. I've never thought of judging simple systems as good/moral, and complex systems as bad/immoral. It just makes no sense to me.

I judge morality of decisions, based on the calculus behind them. E.G., if you said you believed Boeing consciously made a decision they expected would kill people and chose to do it anyway because they'd profit, I'd consider that very bad, and would understand the judgement of "reprehensible". So...

Yes they have cobbled together a fix for the Frankenstein and it will rise. But 346 people are dead. My argument is that the proximate cause should not be substituted for the fundamental cause. The proximate cause was bad design verification and validation; the fundamental cause was choosing profit over good design practice. The proximate cause mistake is regrettable but the fundamental cause mistake is reprehensible.

Ahh -- so that's it: it's not about complexity, it's about a choice of profit over good design practice. Yes, that I can see. I don't know that we have a good handle on the details of the process, but I understand it is possible to imagine they knowingly cut corners or ignored clear signs of an issue. If that's true, that would be really bad.

But I don't know that we really know what went into the decision-making and I prefer to make positive assumptions where I don't know. And here, complacency and impatience would explain it. I judge this way in part because I'm sure they would know that a plane with a significant/fundamental flaw would likely crash and would likely undermine any profit motive. And because we've seen complacency and performance pressure (similar to, but not quite the same as a profit motive) in action before. So for now I choose to believe that the decisions were made with the expectation that they would be unlikely to substantially impact safety.

So I don't necessarily agree, mostly because I just don't know the details of the motivations/decision-making, but at least now I understand it.

 

[PeterDonis]

isn't this speculation that if the FAA had taken a closer look at the system they would have mandated pilot training go along with it?

No. At least, it's not speculation on my part. I'm just going by what we now know to be the primary driver of MCAS within Boeing: the desire to avoid pilot retraining, since they believed (correctly, as far as I can tell) that their airline customers would not buy the plane if they had to pay to retrain their pilots, and Boeing could not afford to pay for the pilot retraining themselves in order to sweeten the deal so the airlines would accept it.

I don't think it's just "pilot retraining", but rather the type rating that is the issue.

From what I understand, the cost of a new type rating would have been far less than the cost of pilot retraining. So even if a new type rating would have been required, I don't think that requirement was the primary driver of Boeing's thinking.

 

[FactChecker]

Sigh, long threads are tedious because things posted earlier may be forgotten.Earlier in this thread this was discussed. The Max had fully redundant A and B strings, with manual switching between them. In the accident planes, if the pilots had switched from A to B, the crashes may have been avoided.

Neither system had the necessary redundancy designed into that system. It is wrong to expect the pilot to know when to switch when the display that he needed to know what was going on was removed. Giving so much authority to a non-redundant system that even ignored contrary pilot input was especially unwise.

The complaint is that A string has one AOA sensor, and B has one AOA sensor, but some people complain that both strings should have access to both sensors without manual switching.

Absolutely!

In that sense, A and B strings would no longer be fully independent. Cross-connections between strings introduce mutual dependencies and new kinds of common mode failures.

Boeing's design can be criticized, but it is unfair to characterize is as lack of redundancy.

If a flight-critical system is given full authority, there should be cross-comparisons of the sensors and logic for a discrepancy. Some airplanes where the flight control is so flight-critical has a third flight control as a tie-breaker and even has a fourth flight control system as a back-up. I admit that designing such a system is a lot more work, but that is what a flight-critical system with full authority requires -- especially if it is going to over-ride contrary pilot inputs.

 

[russ_watters]

From what I understand, the cost of a new type rating would have been far less than the cost of pilot retraining.

I think you have it backwards. Or, rather, "pilot retraining" can be a little retraining whereas a new type rating is a lot of retraining.

 

[PeterDonis]

the option to redesign the landing gear to allow higher ground clearance

IIRC this was a nonstarter because airline customers would have had to rework jetways and their maintenance infrastructure, all of which were designed for the 737's existing ground clearance. For Southwest, in particular, I can imagine that cost would have been a deal breaker; their business model relies heavily on fast turnaround, which in turn relies on every piece of that turnaround being fine tuned for optimum efficiency around the existing 737 footprint and ground clearance.

 

[PeterDonis]

I think you have it backwards.

Possibly I do; it's been a while since I looked at this and I may be misremembering things.

 

[russ_watters]

Possibly I do; it's been a while since I looked at this and I may be misremembering things.

From the wiki linked in my prior post:

In the U.S., the MAX shares a compatible type rating throughout the Boeing 737 series.[28] The impetus for Boeing to build the 737 MAX was serious competition from the Airbus A320neo, which was a threat to win a major order for aircraft from American Airlines, a traditional customer for Boeing airplanes.[29] Boeing decided to update its 737, designed in the 1960s, rather than designing a clean sheet aircraft, which would have cost much more and taken years longer. Boeing's goal was to ensure the 737 MAX would not need a new type rating, which would require significant additional pilot training, adding unacceptably to the overall cost of the airplane for customers.

 

[anorlunda]

IIRC this was a nonstarter because airline customers would have had to rework jetways and their maintenance infrastructure, all of which were designed for the 737's existing ground clearance.

I find that hard to believe because Boeing was planning the 737-MAX-10 with the higher gear. If you are correct, the 737-MAX-9 [edit: with higher gear] would be a financial disaster but the 737-MAX-10 would be fine. Do you have a source?

I don't have any data on how much higher the gear would be. 20 cm? 1 m?

The jetways I see have vertical adjustment. They have painted calibration marks for the correct jetway height for 737, DC9, AIRBUS320, and so on. So I would be surprised if they must be redesigned to accommodate a higher 737-MAX-9 or a 737-MAX-10.

 

[PeterDonis]

Boeing was planning the 737-MAX-10 with the higher gear.

Yes, it was. However, according to Wikipedia [1], the higher gear for the MAX 10 was driven by the need to move the rotation point aft because of the longer fuselage, and did not change anything about the engine configuration relative to the fuselage, which is what causes the pitch up moment at higher angle of attack. So what Boeing did on the MAX 10 does not remove the need for MCAS or something like it.

[1] https://en.wikipedia.org/wiki/Boeing_737_MAX#737_MAX_10

I don't have any data on how much higher the gear would be.

9.5 inches according to the article linked above.

 

[mfb]

Concerning the discussion about modifying an old design: That's exactly what Airbus did - successfully. Take an A320 from the 1980s (not 50 years, okay, but over 30), change the engine. It worked for Airbus because they had enough space to mount the engine at the same place as before. The problem with the 737 MAX was not the old design, it was the engine not fitting to that old design.

 

[artis]

@mfb correct, and add a heavy dose of corporate greed and financial pressure on top of that.

Maybe one can fix an MCAS system for a plane but one will never be able to fix financial greed, there are no patches for that sort of thing and there are no sensors for it to begin with...Engineers typically work best when they are left alone to master their area of expertise. We normally don't hear about financial experts and share holders making decisions on parts of a particle accelerator for example and rightly so because then scientists could get nowhere, then again in more commercial types of business we see a lot of compromise between what would be good and what "fits the bill".

 

[Dullard]

The engineering of commercial products (those that have to compete for customers) always requires consideration of the economics. A competent 'Chief Engineer' should have identified this issue and insisted on a more robust (and probably costly) system. If you believe that engineers ever get to do exactly what they think is 'best,' you're probably a physicist.

 

[FactChecker]

The engineering of commercial products (those that have to compete for customers) always requires consideration of the economics. A competent 'Chief Engineer' should have identified this issue and insisted on a more robust (and probably costly) system. If you believe that engineers ever get to do exactly what they think is 'best,' you're probably a physicist.

If it was totally left up to engineers (not to denigrate engineers, of which I am one), some engineers would be immediately sure that they had designed a perfect system. Other engineers would be so cautious that the plane would never leave the drafting table. And the end result would be decided by a fistfight.

 

[nsaspook]

While buggy and badly designed MCAS IMO wasn't the root cause of why those two planes crashed. The root cause was a lack of training on quickly, effectively diagnosing and correcting a condition of run-away trim. The run-away trim memory items existed long before MCAS and were effective in saving the plane and lives in cases where MCAS and other systems misbehaved in the past even if the pilots didn't know MCAS was installed. Sure it's a very good thing the probability of MCAS as the source has been reduced but maybe it's more important pilots are being trained and retested on how to fly the plane under these confusing conditions.

Run-away trim memory items and the proper methods to recognize and handle it early is what's really been 'fixed' here.


1:10

 

[PeterDonis]

The root cause was a lack of training on quickly, effectively diagnosing and correcting a condition of run-away trim.

I believe this was discussed earlier in this thread (quite a while ago now since the thread was dormant for a while). My recollection of the TL/DR of that discussion is: first, the symptoms of MCAS failure are not the same as the symptoms of a normal runaway trim event; and second, the standard action pilots were trained to take for runaway trim does not disable MCAS; disabling MCAS requires a more complicated series of actions that no pilots were ever trained to carry out.

 

[PeterDonis]

Here is the FAA's statement on the rescission of the emergency order that prohibited the 737 MAX from flying:

https://www.faa.gov/news/updates/?newsId=93206

It includes links to relevant FAA documents.

 

[PeterDonis]

buggy and badly designed MCAS IMO wasn't the root cause of why those two planes crashed

The changes that were made to the flight control software, as described in the FAA's updated Airworthiness Directive, do not seem to me to support this assertion. Key changes that were made (pp. 6-7) include:

MCAS can only activate based on inputs from both AoA sensors, not a single one.

The inputs from the two AoA sensors must be compared, and if they differ significantly, the speed trim system, which includes MCAS, is disabled for the remainder of the flight (and a light illuminates in the cockpit to indicate this).

Only one MCAS activation is permitted per high AoA event.

The control authority of MCAS is limited such that, even when MCAS is commanding the maximum change it is allowed to the horizontal stabilizer, the pilot can still control pitch using the control column, without having to make any electric or manual stabilizer trim inputs.

The fact that those changes were required indicates to me that the errors in the control software that those changes are correcting were part of the root cause of the two crashes.

Also note that the updated pilot training required for the 737 MAX now includes training in how to recognize an AoA sensor failure and how to get the plane's trim back into a reasonable range before disabling the electric trim system in the event of an AoA sensor failure that triggers an erroneous MCAS activation.

 

[FactChecker]

It is clear from the FAA analysis that the design was riddled with fundamental errors that needed to be corrected. Training is certainly one thing to correct, but the design violated many basic principles that are the first thing to correct before pilot training is even looked at. The design mistakes were inconceivable.

 

[nsaspook]

I believe this was discussed earlier in this thread (quite a while ago now since the thread was dormant for a while). My recollection of the TL/DR of that discussion is: first, the symptoms of MCAS failure are not the same as the symptoms of a normal runaway trim event; and second, the standard action pilots were trained to take for runaway trim does not disable MCAS; disabling MCAS requires a more complicated series of actions that no pilots were ever trained to carry out.

I have to disagree.

The normal memory checklist for a runaway trim event is to disable the trim power by flipping switches. This disables any possible electrical movement command including MCAS.

 

[PeterDonis]

The normal memory checklist for a runaway trim event is to disable the trim power by flipping switches. This disables any possible electrical movement command including MCAS.

It's not that simple.

First, disabling the electric trim system completely, which disables MCAS as well, means you have to put the trim back where it's supposed to be using the mechanical trim wheel. That can be prohibitively difficult or even impossible once MCAS has put the trim far enough in the wrong direction--MCAS before the changes now being implemented had enough control authority to put the trim in a place where it is physically impossible to readjust it using the mechanical trim wheel. (And in such a position the pilot also cannot exert enough force on the control column to have the needed pitch authority.)

Second, disabling just automatic electric trim while leaving the manual electric trim powered, so you can put the trim back where it belongs using the manual electric trim system, does not disable MCAS. So you get into repeated cycles of MCAS putting the trim out of whack, using the manual electric trim system to readjust it, and then MCAS putting it out of whack again. The only way out of this loop is to use the manual electric trim system to put the trim back where it belongs, and then immediately shut off electric trim completely, so you're now restricted to the mechanical trim wheel for the remainder of the flight. That is what the updated pilot training now trains pilots to do in the event of an erroneous MCAS trim adjustment; but the previous pilot training did not train them to do that.

Both of these issues were factors in the crashes. (And all of this has been well discussed previously in this thread, though of course it's been a while now.)

 

[nsaspook]

It's not that simple.

First, disabling the electric trim system completely, which disables MCAS as well, means you have to put the trim back where it's supposed to be using the mechanical trim wheel. That can be prohibitively difficult or even impossible once MCAS has put the trim far enough in the wrong direction--MCAS before the changes now being implemented had enough control authority to put the trim in a place where it is physically impossible to readjust it using the mechanical trim wheel.

Second, disabling just automatic electric trim while leaving the manual electric trim powered, so you can put the trim back where it belongs using the manual electric trim system, does not disable MCAS. So you get into repeated cycles of MCAS putting the trim out of whack, using the manual electric trim system to readjust it, and then MCAS putting it out of whack again. The only way out of this loop is to use the manual electric trim system to put the trim back where it belongs, and then immediately shut off electric trim completely, so you're now restricted to the mechanical trim wheel for the remainder of the flight. That is what the updated pilot training now trains pilots to do in the event of an erroneous MCAS trim adjustment; but the previous pilot training did not train them to do that.

Both of these issues were factors in the crashes.

I agree that early detection of the problem is the key, ie training. The old procedure worked to disable MCAS in a recoverable mode if you didn't allow the trim to move the jack-screw to extreme locations that required beyond human effort.

In the Lion air case the day before the fatal crash a crew did the Trim runaway memory item correctly, MCAS was disconnected when the trim power was cut, plane landed safely.
https://en.wikipedia.org/wiki/Lion_Air_Flight_610

Passengers recounted that the aircraft had suffered an engine problem and were told not to board it as engineers tried to fix the problem. While the aircraft was en route to Jakarta, it had problems maintaining a constant altitude, with passengers stating that it was like "a roller-coaster ride."[118] The chief executive officer of Lion Air, Edward Sirait, said the aircraft had a "technical issue" on Sunday night, but this had been addressed in accordance with maintenance manuals issued by the manufacturer. Engineers had declared that the aircraft was ready for takeoff on the morning of the accident.[119][120] Information later emerged that a third pilot was on the flight to Jakarta and told the crew to cut power to the stabilizer trim motors which fixed the problem. This method is a standard memory item in the 737 checklist.[121] Subsequently, the National Transportation Safety Committee confirmed the presence of an off-duty Boeing 737 MAX 8 qualified pilot in the cockpit but did not confirm the role of the pilot in fixing the problem, and denied that there was any recording of the previous flight in the CVR of Lion Air Flight 610.[122]

 

[PeterDonis]

The old procedure worked if you didn't allow the trim to move the jack-screw to extreme locations that required beyond human effort.

Yes, but having that happen was a matter of luck. See below.

In the Lion air case the day before the fatal crash a crew did the Trim runaway memory item correctly

But only because there was an off duty pilot sitting in the jump seat in the cockpit, who, not being distracted by all the other stuff that was going on in the cockpit (note that another item now being added in the 737 MAX pilot training is how to deal with multiple warnings in the cockpit all going off at the same time), was able to figure out what to do and told the crew to do it.

 

[nsaspook]

Yes, but having that happen was a matter of luck. See below.
But only because there was an off duty pilot sitting in the jump seat in the cockpit, who, not being distracted by all the other stuff that was going on in the cockpit (note that another item now being added in the 737 MAX pilot training is how to deal with multiple warnings in the cockpit all going off at the same time), was able to figure out what to do and told the crew to do it.

That's why I included the example. It was possible to disable MCAS even it you didn't know the system existed but knowing it existed and training on how to handing its unique signature of failure might have saved both flights.

 

[PeterDonis]

a crew did the Trim runaway memory item correctly

Also note that, in the updated pilot training for the 737 MAX, this item is now different: before shutting off the electric trim system, you now have to check to make sure the trim is close enough to where it should be for mechanical adjustment, and if it isn't, you have to use the manual electric trim system to put it there. So saying that "the old procedure worked" is, IMO, a misstatement; the old procedure did not work as the pilots were trained to do it, because it ignored the possibility of the trim being in a condition where mechanical adjustment was not possible. The reason for that was that the old procedure was developed before MCAS existed, and before MCAS existed, there was not a possibility of the automatic electric trim system putting the trim in a place where mechanical adjustment was not possible; without MCAS that system cannot do that. So adding MCAS should have originally included adding that extra check and operation to the procedure which is now added.

 

[nsaspook]

Also note that, in the updated pilot training for the 737 MAX, this item is now different: before shutting off the electric trim system, you now have to check to make sure the trim is close enough to where it should be for mechanical adjustment, and if it isn't, you have to use the manual electric trim system to put it there. So saying that "the old procedure worked" is, IMO, a misstatement; the old procedure did not work as the pilots were trained to do it, because it ignored the possibility of the trim being in a condition where mechanical adjustment was not possible. The reason for that was that the old procedure was developed before MCAS existed, and before MCAS existed, there was not a possibility of the automatic electric trim system putting the trim in a place where mechanical adjustment was not possible; without MCAS that system cannot do that. So adding MCAS should have originally included adding that extra check and operation to the procedure which is now added.

I mainly agree but that all assumes you still have a functional electrical system to manual trim, switches fail, wires short, motors jam. There was always the possibility of electric trim system putting the trim in a place where mechanical adjustment was not possible long before MCAS.

http://www.b737.org.uk/runawaystab.htm#rc

 

[FactChecker]

The original design would send the plane into a dive due to the AOA signal WITHOUT EVEN CHECKING IF THERE WAS AN AOA MISCOMPARE.
It had complete authority that the pilot could not overcome.
It was persistent and would turn itself back on, giving itself more control time than it gave to the pilot.
It removed the needed AOA miscompare indication from the pilot displays unless they had paid an additional amount for it.
These are all terrible design decisions. The fact that there was also a training issue should not be used as an excuse for these design mistakes. The corrective actions take care of them all and will make the plane much, much safer. That is what a design should do.

 

[PeterDonis]

There was always the possibility of electric trim system putting the trim in a place where mechanical adjustment was not possible long before MCAS.

But if the electric trim system is failed, you can't use it to get out of such a situation. That's not what we're talking about. We're talking about a situation where the electric trim system is working, so it can be used to get out of such a situation--but the only way to get into such a situation with a working electric trim system is MCAS.

 

[nsaspook]

But if the electric trim system is failed, you can't use it to get out of such a situation. That's not what we're talking about. We're talking about a situation where the electric trim system is working, so it can be used to get out of such a situation--but the only way to get into such a situation with a working electric trim system is MCAS.

What about the auto-pilot (a separate system from MCAS)? It controls trim too and is on the run-away trim memory checklist.

Training is not a excuse for bad engineering. Good operations means being prepared with proper training for the unlikely and practicing the impossible in training scenarios.

 

[PeterDonis]

What about the auto-pilot (a separate system from MCAS)?

What about it?

Training is not a excuse for bad engineering.

Agreed. But you were arguing that bad engineering was not the root cause of the 737 MAX crashes. I don't see how that follows from the fact that the training was also bad. Both were bad, and both were contributing root causes of the crashes.

Good operations means being prepared with proper training for the unlikely and practicing the impossible in training scenarios.

Yes, and the 737 MAX training prior to these new changes did not do that; it didn't even tell pilots that MCAS existed. How can pilots be expected to properly understand what the airplane is doing if they don't even know of the existence of an important system?

 

[nsaspook]

What about it?
Agreed. But you were arguing that bad engineering was not the root cause of the 737 MAX crashes. I don't see how that follows from the fact that the training was also bad. Both were bad, and both were contributing root causes of the crashes.
Yes, and the 737 MAX training prior to these new changes did not do that; it didn't even tell pilots that MCAS existed. How can pilots be expected to properly understand what the airplane is doing if they don't even know of the existence of an important system?

The 737 auto-pilot is on the run-away trim checklist because MCAS is not the only thing that can cause the trim system to move to a mechanically hard to recover position with a working electric trim system.

Yes. It's because MCAS was a bandaid to cover some MAX extreme flight control issues that would have required type training. The reason MCAS is the 737 MAX is to eliminate a training requirement. It wasn't needed to fly the plane safely.

Agree.

 

[Ivan Seeking]

I believe this was discussed earlier in this thread (quite a while ago now since the thread was dormant for a while). My recollection of the TL/DR of that discussion is: first, the symptoms of MCAS failure are not the same as the symptoms of a normal runaway trim event; and second, the standard action pilots were trained to take for runaway trim does not disable MCAS; disabling MCAS requires a more complicated series of actions that no pilots were ever trained to carry out.

I have some inside knowledge of this event. My understanding is that ultimately it came down to two misplaced lines of code . It is being called by some the most expensive programming error in history.

 

[nsaspook]

I have some inside knowledge of this event. My understanding is that ultimately it came down to two misplaced lines of code . It is being called by some the most expensive programming error in history.

All of this death, cost and work for a system to adjust the pilot column pull to give the MAX the flying feel of older 737 models.

A classic example of how shortcuts become disastrous.