Dominic Gates, Steve Miletich, Mike Baker and Lewis Kamb of The Seattle Times

By Dominic Gates

As Boeing hustled in 2015 to catch up to Airbus and certify its new 737 MAX, Federal Aviation Administration (FAA) managers pushed the agency’s safety engineers to delegate safety assessments to Boeing itself, and to speedily approve the resulting analysis.

But the original safety analysis that Boeing delivered to the FAA for a new flight control system on the MAX — a report used to certify the plane as safe to fly — had several crucial flaws.

That flight control system, called MCAS (Maneuvering Characteristics Augmentation System), is now under scrutiny after two crashes of the jet in less than five months resulted in the FAA’s March 13 order to ground the plane.

Current and former engineers directly involved with the evaluations or familiar with the document shared details of Boeing’s “System Safety Analysis” of MCAS, which The Seattle Times confirmed.

The safety analysis:

• Understated the power of the new flight control system, which was designed to swivel the horizontal tail to push the nose of the plane down to avert a stall. When the planes later entered service, MCAS was capable of moving the tail more than four times farther than was stated in the initial safety analysis document.
• Failed to account for how the system could reset itself each time a pilot responded, thereby missing the potential impact of the system repeatedly pushing the airplane’s nose downward.
• Assessed a failure of the system as one level below “catastrophic.” But even that “hazardous” danger level should have precluded activation of the system based on input from a single sensor — and yet that’s how it was designed.

The people who spoke to The Seattle Times and shared details of the safety analysis all spoke on condition of anonymity to protect their jobs at the FAA and other aviation organizations.

Both Boeing and the FAA were informed of the specifics of this story and were asked for responses 11 days ago, before the second crash of a 737 MAX on March 10.

Late on the 15th, the FAA said it followed its standard certification process on the MAX. Citing a busy week, a spokesman said the agency was “unable to delve into any detailed inquiries.”

Boeing responded March 16 with a statement that “the FAA considered the final configuration and operating parameters of MCAS during MAX certification, and concluded that it met all certification and regulatory requirements.”

Adding that it is “unable to comment … because of the ongoing investigation” into the crashes, Boeing did not respond directly to the detailed description of the flaws in MCAS certification, beyond saying that “there are some significant mischaracterizations.”

Several technical experts inside the FAA said October’s Lion Air crash, where the MCAS has been clearly implicated by investigators in Indonesia, is only the latest indicator that the agency’s delegation of airplane certification has gone too far, and that it’s inappropriate for Boeing employees to have so much authority over safety analyses of Boeing jets.

“We need to make sure the FAA is much more engaged in failure assessments and the assumptions that go into them,” said one FAA safety engineer.

Certifying a new flight control system

Going against a long Boeing tradition of giving the pilot complete control of the aircraft, the MAX’s new MCAS automatic flight control system was designed to act in the background, without pilot input.

It was needed because the MAX’s much larger engines had to be placed farther forward on the wing, changing the airframe’s aerodynamic lift.

Designed to activate automatically only in the extreme flight situation of a high-speed stall, this extra kick downward of the nose would make the plane feel the same to a pilot as the older-model 737s.

Boeing engineers authorized to work on behalf of the FAA developed the System Safety Analysis for MCAS, a document which in turn was shared with foreign air-safety regulators in Europe, Canada and elsewhere in the world.

The document, “developed to ensure the safe operation of the 737 MAX,” concluded that the system complied with all applicable FAA regulations.

Yet black box data retrieved after the Lion Air crash indicates that a single faulty sensor — a vane on the outside of the fuselage that measures the plane’s “angle of attack,” the angle between the airflow and the wing — triggered MCAS multiple times during the deadly flight, initiating a tug of war as the system repeatedly pushed the nose of the plane down and the pilots wrestled with the controls to pull it back up, before the final crash.

When announcing the grounding of the 737 MAX, the FAA cited similarities in the flight trajectory of the Lion Air flight and the crash of Ethiopian Airlines Flight 302.

Investigators also found the Ethiopian plane’s jackscrew, a part that moves the horizontal tail of the aircraft, and it indicated that the jet’s horizontal tail was in an unusual position — with MCAS as one possible reason for that.

Investigators are working to determine if MCAS could be the cause of both crashes.

Delegated to Boeing

The FAA, citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes.

Early on in certification of the 737 MAX, the FAA safety engineering team divided up the technical assessments that would be delegated to Boeing versus those they considered more critical and would be retained within the FAA.

But several FAA technical experts said in interviews that as certification proceeded, managers prodded them to speed the process. Development of the MAX was lagging nine months behind the rival Airbus A320neo. Time was of the essence for Boeing.

A former FAA safety engineer who was directly involved in certifying the MAX said that halfway through the certification process, “we were asked by management to re-evaluate what would be delegated. Management thought we had retained too much at the FAA.”

“There was constant pressure to re-evaluate our initial decisions,” the former engineer said. “And even after we had reassessed it … there was continued discussion by management about delegating even more items down to the Boeing Company.”

Even the work that was retained, such as reviewing technical documents provided by Boeing, was sometimes curtailed.

“There wasn’t a complete and proper review of the documents,” the former engineer added. “Review was rushed to reach certain certification dates.”

When time was too short for FAA technical staff to complete a review, sometimes managers either signed off on the documents themselves or delegated their review back to Boeing.

“The FAA managers, not the agency technical experts, have final authority on delegation,” the engineer said.

Inaccurate limit

In this atmosphere, the System Safety Analysis on MCAS, just one piece of the mountain of documents needed for certification, was delegated to Boeing.

The original Boeing document provided to the FAA included a description specifying a limit to how much the system could move the horizontal tail — a limit of 0.6 degrees, out of a physical maximum of just less than 5 degrees of nose-down movement.

That limit was later increased after flight tests showed that a more powerful movement of the tail was required to avert a high-speed stall, when the plane is in danger of losing lift and spiraling down.

The behavior of a plane in a high angle-of-attack stall is difficult to model in advance purely by analysis and so, as test pilots work through stall-recovery routines during flight tests on a new airplane, it’s not uncommon to tweak the control software to refine the jet’s performance.

After the Lion Air Flight 610 crash, Boeing for the first time provided to airlines details about MCAS. Boeing’s bulletin to the airlines stated that the limit of MCAS’s command was 2.5 degrees.

That number was new to FAA engineers who had seen 0.6 degrees in the safety assessment.

“The FAA believed the airplane was designed to the 0.6 limit, and that’s what the foreign regulatory authorities thought, too,” said an FAA engineer. “It makes a difference in your assessment of the hazard involved.”

The higher limit meant that each time MCAS was triggered, it caused a much greater movement of the tail than was specified in that original safety analysis document.

The former FAA safety engineer who worked on the MAX certification, and a former Boeing flight controls engineer who worked on the MAX as an authorized representative of the FAA, both said that such safety analyses are required to be updated to reflect the most accurate aircraft information following flight tests.

“The numbers should match whatever design was tested and fielded,” said the former FAA engineer.

But both said that sometimes agreements were made to update documents only at some later date.

“It’s possible the latest numbers wouldn’t be in there, as long as it was reviewed and they concluded the differences wouldn’t change the conclusions or the severity of the hazard assessment,” said the former Boeing flight controls engineer.

If the final safety analysis document was updated in parts, it certainly still contained the 0.6 limit in some places and the update was not widely communicated within the FAA technical evaluation team.

“None of the engineers were aware of a higher limit,” said a second current FAA engineer.

The discrepancy over this number is magnified by another element in the System Safety Analysis: The limit of the system’s authority to move the tail applies each time MCAS is triggered. And it can be triggered multiple times, as it was on the Lion Air flight.

One current FAA safety engineer said that every time the pilots on the Lion Air flight reset the switches on their control columns to pull the nose back up, MCAS would have kicked in again and “allowed new increments of 2.5 degrees.”

“So once they pushed a couple of times, they were at full stop,” meaning at the full extent of the tail swivel, he said.

Peter Lemme, a former Boeing flight controls engineer who is now an avionics and satellite-communications consultant, said that because MCAS reset each time it was used, “it effectively has unlimited authority.”

Swiveling the horizontal tail, which is technically called the stabilizer, to the end stop gives the airplane’s nose the maximum possible push downward.

“It had full authority to move the stabilizer the full amount,” Lemme said. “There was no need for that. Nobody should have agreed to giving it unlimited authority.”

On the Lion Air flight, when the MCAS pushed the jet’s nose down, the captain pulled it back up, using thumb switches on the control column. Still operating under the false angle-of-attack reading, MCAS kicked in each time to swivel the horizontal tail and push the nose down again.

The black box data released in the preliminary investigation report shows that after this cycle repeated 21 times, the plane’s captain ceded control to the first officer. As MCAS pushed the nose down two or three times more, the first officer responded with only two short flicks of the thumb switches.

At a limit of 2.5 degrees, two cycles of MCAS without correction would have been enough to reach the maximum nose-down effect.

In the final seconds, the black box data shows the captain resumed control and pulled back up with high force. But it was too late. The plane dived into the sea at more than 500 miles per hour.

System failed on a single sensor

The bottom line of Boeing’s System Safety Analysis with regard to MCAS was that, in normal flight, an activation of MCAS to the maximum assumed authority of 0.6 degrees was classified as only a “major failure,” meaning that it could cause physical distress to people on the plane, but not death.

In the case of an extreme maneuver, specifically when the plane is in a banked descending spiral, an activation of MCAS was classified as a “hazardous failure,” meaning that it could cause serious or fatal injuries to a small number of passengers. That’s still one level below a “catastrophic failure,” which represents the loss of the plane with multiple fatalities.

The former Boeing flight controls engineer who worked on the MAX’s certification on behalf of the FAA said that whether a system on a jet can rely on one sensor input, or must have two, is driven by the failure classification in the system safety analysis.

He said virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the “major failure” requirement, which is that the probability of a failure must be less than one in 100,000. Such systems are therefore typically allowed to rely on a single input sensor.

But when the consequences are assessed to be more severe, with a “hazardous failure” requirement demanding a more stringent probability of one in 10 million, then a system typically must have at least two separate input channels in case one goes wrong.

Boeing’s System Safety Analysis assessment that the MCAS failure would be “hazardous” troubles former flight controls engineer Lemme because the system is triggered by the reading from a single angle-of-attack sensor.

“A hazardous failure mode depending on a single sensor, I don’t think passes muster,” said Lemme.

Like all 737s, the MAX actually has two of the sensors, one on each side of the fuselage near the cockpit. But the MCAS was designed to take a reading from only one of them.

Lemme said Boeing could have designed the system to compare the readings from the two vanes, which would have indicated if one of them was way off.

Alternatively, the system could have been designed to check that the angle-of-attack reading was accurate while the plane was taxiing on the ground before takeoff, when the angle of attack should read zero.

“They could have designed a two-channel system. Or they could have tested the value of angle of attack on the ground,” said Lemme. “I don’t know why they didn’t.”

The black box data provided in the preliminary investigation report shows that readings from the two sensors differed by some 20 degrees not only throughout the flight but also while the airplane taxied on the ground before takeoff.

No training, no information

After the Lion Air crash, 737 MAX pilots around the world were notified about the existence of MCAS and what to do if the system is triggered inappropriately.

Boeing insists that the pilots on the Lion Air flight should have recognized that the horizontal stabilizer was moving uncommanded, and should have responded with a standard pilot checklist procedure to handle what’s called “stabilizer runaway.”

If they’d done so, the pilots would have hit cutoff switches and deactivated the automatic stabilizer movement.

Boeing has pointed out that the pilots flying the same plane on the day before the crash experienced similar behavior to Flight 610 and did exactly that: They threw the stabilizer cutoff switches, regained control and continued with the rest of the flight.

However, pilots and aviation experts say that what happened on the Lion Air flight doesn’t look like a standard stabilizer runaway, because that is defined as continuous uncommanded movement of the tail.

On the accident flight, the tail movement wasn’t continuous; the pilots were able to counter the nose-down movement multiple times.

In addition, the MCAS altered the control column response to the stabilizer movement. Pulling back on the column normally interrupts any stabilizer nose-down movement, but with MCAS operating that control column function was disabled.

These differences certainly could have confused the Lion Air pilots as to what was going on.

Since MCAS was supposed to activate only in extreme circumstances far outside the normal flight envelope, Boeing decided that 737 pilots needed no extra training on the system — and indeed that they didn’t even need to know about it. It was not mentioned in their flight manuals.

That stance allowed the new jet to earn a common “type rating” with existing 737 models, allowing airlines to minimize training of pilots moving to the MAX.

Dennis Tajer, a spokesman for the Allied Pilots Association at American Airlines, said his training on moving from the old 737 NG model cockpit to the new 737 MAX consisted of little more than a one-hour session on an iPad, with no simulator training.

Minimizing MAX pilot transition training was an important cost saving for Boeing’s airline customers, a key selling point for the jet, which has racked up more than 5,000 orders.

The company’s website pitched the jet to airlines with a promise that “as you build your 737 MAX fleet, millions of dollars will be saved because of its commonality with the Next-Generation 737.”

In the aftermath of the crash, officials at the unions for both American and Southwest Airlines pilots criticized Boeing for providing no information about MCAS, or its possible malfunction, in the 737 MAX pilot manuals.

An FAA safety engineer said the lack of prior information could have been crucial in the Lion Air crash.

Boeing’s safety analysis of the system assumed that “the pilots would recognize what was happening as a runaway and cut off the switches,” said the engineer. “The assumptions in here are incorrect. The human factors were not properly evaluated.”

On March 11, before the grounding of the 737 MAX, Boeing outlined “a flight control software enhancement for the 737 MAX,” that it’s been developing since soon after the Lion Air crash.

According to a detailed FAA briefing to legislators, Boeing will change the MCAS software to give the system input from both angle-of-attack sensors.

It will also limit how much MCAS can move the horizontal tail in response to an erroneous signal. And when activated, the system will kick in only for one cycle, rather than multiple times.

Boeing also plans to update pilot training requirements and flight crew manuals to include MCAS.

These proposed changes mirror the critique made by the safety engineers in this story. They had spoken to The Seattle Times before the Ethiopian crash.

The FAA said it will mandate Boeing’s software fix in an airworthiness directive no later than April.

Facing legal actions brought by the families of those killed, Boeing will have to explain why those fixes were not part of the original system design. And the FAA will have to defend its certification of the system as safe.

This story has been updated to put dates on references to days of the week following the second crash.

By Dominic Gates and Mike Baker

In 2016, as Boeing raced to get the 737 MAX certified by the Federal Aviation Administration (FAA), a senior company engineer whose job was to act on behalf of the FAA balked at Boeing management demands for less stringent testing of the fire-suppression system around the jet’s new LEAP engines.

That June he convened a meeting of all the certification engineers in his unit, who collectively agreed with his assessment. Management initially rejected their position, and only after another senior engineer from outside the MAX program intervened did managers finally agree to beef up the testing to a level the engineer could accept, according to two people familiar with the matter.

But his insistence on a higher level of safety scrutiny cost Boeing time and money.

Less than a month after his peers had backed him, Boeing abruptly removed him from the program even before conducting the testing he’d advocated.

The episode underscores what The Seattle Times found after a review of documents and interviews with more than a dozen current and former Boeing engineers who have been involved in airplane certification in recent years, including on the 737 MAX: Many engineers, employed by Boeing while officially designated to be the FAA’s eyes and ears, faced heavy pressure from Boeing managers to limit safety analysis and testing so the company could meet its schedule and keep down costs.

That pressure increased when the FAA stopped dealing directly with those designated employees — called “Authorized Representatives” or ARs — and let Boeing managers determine what was presented to the regulatory agency.

“The ARs have nobody supporting them. Nobody has their backs,” said one former Authorized Representative who worked on the 737 MAX and who provided details of the engineer’s removal from the program. “The system is absolutely broken.”

FAA-designated oversight engineers are supposed to enjoy protection from management pressure. Removing one who proves a stickler for safety regulations will inevitably produce a chilling effect on others who see the consequences of being too rigid about safety concerns, said John Goglia, former member of the National Transportation Safety Board (NTSB).

“It negates the whole system,” said Goglia. “The FAA should have come down on that really hard.”

Following two deadly 737 MAX crashes off the coast of Indonesia and in Ethiopia that killed 346 people, and the subsequent grounding of the airplane worldwide, the certification of the jet has come under intense scrutiny, including a slew of lawsuits, congressional hearings and a criminal investigation.

None of the people interviewed were involved in certifying the Maneuvering Characteristics Augmentation System, the flight-control software implicated in the two crashes. But one area of scrutiny is sure to be the delegated system under which Boeing employees, paid by the company but acting as FAA designees, did the detailed certification work. It may slow down plans by the FAA and Boeing for a future certification regimen that would further erode the FAA’s oversight.

Boeing, in a statement responding to Seattle Times questions, said that FAA procedures, including regular, FAA-mandated training, “ensure Boeing employees serving in this capacity act independently on behalf of the FAA.”

It added that “there are processes in place to carefully evaluate any concerns regarding the AR’s ability to act independently.” The company declined to comment on individual cases cited in this story.

Yet as the FAA has increasingly delegated certification tasks to Boeing itself, it’s also made changes to the reporting structure that leave its designees to fend for themselves inside the company.

While a few former employees involved in certifications said they handled the pressure as a regular part of the job, others described the work environment as hostile, focused on achieving FAA approval within schedule and cost targets. Some of those workers spoke on condition of anonymity to protect professional relationships or for fear of retribution.

This echoes the findings of a Seattle Times investigation in March of what happened on the FAA side of the MAX certification. Within the FAA, its safety engineers worked under constant pressure from their managers to delegate more and more work to Boeing itself, and to speedily approve the safety assessments the Boeing designees came up with.

On the Boeing side of that process, the removal of the senior engineer acting as an FAA Authorized Rep was an extreme example that highlights the broader negative impact of two changes: The FAA no longer appoints its own ARs, instead leaving that to Boeing. And these designees now rarely interact with the FAA directly, according to former Boeing ARs interviewed by The Times.

They said these changes have stripped them of protection and given managers more opportunity to push for shortcuts.

In a statement, the FAA said it oversees the Boeing certification system “to ensure procedures are followed.” The agency also said it has “received no whistleblower complaints or any other reports … alleging pressure to speed up 737 MAX certification.”

Boeing managers are supposed to undergo “undue pressure” training to ensure that they aren’t crossing boundaries with the FAA’s representatives. And some ARs said that, despite some tensions, their managers were respectful of the role.

Fred Stong, an AR who worked on electrical systems at Boeing, said his experience was that everyone works through differences to reach common ground. He said he was always assertive in his role and didn’t face any problems.

“At no time in my career would anybody dare to pressure me,” Stong said.

Yet the former AR on the MAX said managers overseeing that jet’s certification were “extremely aggressive” about anything that affected the program cost or schedule.

“Managers were pounding on the ARs to get what the company needs in terms of reduced testing,” he said. “If it costs the company time and money, they’d pound on you to change the test design.”

The radical shift from DERs to ARs

Before 2004, those Boeing technical employees who worked safety on behalf of the FAA were called “Designated Engineering Representatives,” or DERs. Though paid by Boeing, they were appointed by the FAA and reported directly to their technical counterparts at the FAA.

What changed since 2004 is that safety engineers, now called Authorized Representatives, are appointed by and report to Boeing managers.

The opaque bureaucratic name for this new structure — Organization Designation Authorization (ODA) — masks the significant change: Instead of having individual Boeing employees authorized as FAA reps, Boeing now has an entire organization within the company so authorized. The individual FAA Authorized Reps — Boeing engineers — report up the chain to their Boeing managers, not the FAA.

A veteran aviation-safety engineer who over the decades worked for long stints as a DER at Boeing and later as a Boeing AR on a variety of projects including the MAX, said there’s “nothing inherently wrong” with the FAA delegating safety certification — provided it retains oversight.

This consultant asked for anonymity to protect his current livelihood doing certification work for multiple aviation companies.

Working as a DER with smaller aviation companies that don’t have an ODA designation, it’s his job to ensure their products comply with all safety regulations. On those projects, he can consult directly with FAA technical people if any problem arises or if he needs advice on what exactly may be required to demonstrate compliance.

“If I need guidance, I call my FAA adviser,” he said. “I’m overseen directly by the FAA. And every year there is a pretty robust audit of my activity before the FAA will delegate me for the following year.”

His experience working as an AR at Boeing and other companies was quite different.

“Under ODA, the FAA no longer manages the people making the compliance findings,” he said. “They never even talk to them.”

And because Boeing appoints the representatives, he said, accountability is severely curtailed. “If the company is happy with their decisions, obviously, they’ll be kept in their jobs.”

Under the old system, “we knew we’d lose our livelihood if we didn’t maintain the integrity of making decisions the way the FAA would do it,” the consultant said. “That check is no longer there.”

The FAA, contradicting the accounts of the former Authorized Representatives interviewed, said that ARs “have frequent interaction and access to FAA personnel to communicate concerns directly.”

However, a copy of one version of the Boeing ODA manual, an internal document labeled proprietary but obtained by The Times, told ARs with concerns about their workload or pressure from managers to first report them to the AR administrator, who is a higher level Boeing manager.

The manual also states that AR performance will be judged in part by whether they are “completing their duties in a timely and cooperative manner.”

It’s a Boeing manager who determines if an individual representative’s performance is sufficiently cooperative, as evidenced by the experience of Mike Levenson, who has worked as an FAA representative at several companies and served in an AR role at Boeing for five years until 2013.

He said that while there’s always a pressure on FAA representatives in an aviation world full of deadlines and cost considerations, most industry managers are able to find a balance to ensure the ARs have independence. He said he didn’t find that to be the case at Boeing.

Levenson worked on certifying aircraft repairs at Boeing and said he certified more than 500 in his time there, though he did not work on the MAX. On three occasions, he declined to certify repairs. The first two times, Levenson said, he got called into a supervisor’s office.

On the third occasion, in June 2013, a proposed repair clearly did not meet all FAA requirements, he said. After he declined to approve it, Levenson said, his manager “told me to go back and find compliance or my contract would not be extended.”

Levenson agreed to do additional work and consulted with other colleagues but still couldn’t certify the repair’s compliance.

“When I reported this to my manager, I was told this was unacceptable and was summarily dismissed the following day,” Levenson said.

The FAA said it has no record of Levenson filing a complaint. Levenson said he talked to the agency but didn’t file anything formally.

MAX inherits 737 legacy issues

The removal from the MAX program of the FAA’s Authorized Rep who insisted upon stricter engine fire-suppression testing is briefly summarized in a February 2017 report obtained by The Seattle Times. The report does not name the engineer, and the two people who described what happened spoke on condition that he not be named.

In the report, prepared by the three unions that represent FAA technical staff, the incident was listed among a long series of problematic decisions made under the current system of delegating FAA certification and oversight to Boeing.

The engineer removed from the program had more than two decades of experience at Boeing doing certification work on behalf of the FAA. Managers transferred him to Boeing’s “Central Engineering” unit, with no particular job description, and appointed as his replacement on the MAX team an engineer with relatively little experience in certification.

Four additional concerns specific to the 737 MAX were listed in the 2017 report. All were related to certification of legacy systems inherited from the earliest 737 models that were found by FAA technical staff to be noncompliant with the latest safety regulations.

These involved a lack of redundancy in the rudder cables; a too-high surface temperature allowed in the fuel tank; insufficient fireproofing around the plane’s auxiliary power unit in the tail; and using high-power wiring to connect to a switch inside the fuel tank.

All these issues were flagged by safety engineers working at the FAA as requiring fixes before the MAX could be certified.

The MAX won certification anyway after managers on the Boeing side of certification insisted that these were non-issues and managers on the FAA side agreed to let it move ahead with these shortcomings unaddressed.

All were waved through by the Boeing ODA and signed off by FAA management, according to the union report.

The FAA, in its statement to The Times, said it ordered the findings to be investigated at the time but said it wouldn’t address the specific items “because of the ongoing investigations into the aircraft’s certification.”

A better oversight structure

When Acting FAA Administrator Dan Elwell appeared before the U.S. Senate subcommittee on aviation in March, he was asked if the FAA could pull the oversight of air safety back in-house instead of delegating it to Boeing and other manufacturers.

“It would require roughly 10,000 more employees and another $1.8 billion for our certification office,” Elwell told the senators.

But that’s assuming the FAA would end delegation of oversight completely and take back all the certification work for a new airplane. That’s impractical, not only for the lack of resources, but also because all the leading-edge technological expertise needed is concentrated inside Boeing and its suppliers.

Many of the FAA’s safety engineers formerly worked for Boeing. But when they leave industry to work for the government, after a few years they inevitably lose touch with the latest innovations.

As the former NTSB member Goglia puts it: “You can’t stay on the pointy end of the arrow and work for the government.”

The former Boeing Authorized Rep who described the current system as “broken” agrees.

“It’s impossible for someone sitting at a desk at the FAA to keep up with the technology,” he said. “Once you step out, it will bypass you really fast.”

Still, he said, there’s no need to contemplate a wholesale removal of delegation from industry. Instead, he said, what’s needed is to have the same Boeing engineers continue to do the safety evaluations, but to have them chosen by and reporting to the FAA — in other words, to revert to the old DER structure of oversight.

The former AR said that worked well because the FAA “was able to see into the design process from the beginning and have direct input as it was developed.”

“I’m not asking for the FAA to add 10,000 engineers,” he said. “Keep the same ARs as today. Just change who they report to, who is overseeing them. That doesn’t mean transferring the work to the FAA.”

John Cox, chief executive of Safety Operating Systems and formerly the top safety official for the Air Line Pilots Association, said that following the accidents and questions raised about how the errant flight control system on the MAX was certified, “there probably needs to be a review of the ODA system.”

“The (older) DER approach worked extremely effectively,” Cox said. “If engineers are working on behalf of the FAA, they should have a direct technical liaison with the FAA.”

And Goglia, the former NTSB member, said the AR system, with these engineers appointed by and reporting to Boeing, may need to be adjusted.

“I like the older system better than Boeing, or any manufacturer, having that kind of control,” Goglia said.

Moving toward complete self-certification

Yet before the MAX crashes, the FAA was heading in exactly the opposite direction: toward more delegation of oversight, with FAA participation reduced to a bare minimum.

A 2012 report to the FAA by a committee co-chaired by a Boeing representative and the FAA’s top aviation safety official, Ali Bahrami, recommended increased delegation of oversight to industry, working toward a “future state” beyond ODA with another deliberately obscure bureaucratic name: Certified Design Organization, or CDO.

If Boeing were to achieve CDO status, its employees could certify their own designs. Employees doing the certification work would not be designees technically working on behalf of the FAA, just Boeing engineers working for Boeing.

This would be true self-certification, but has not yet been implemented.

Levenson said such a shift would increase safety risks for the industry.

“It’s a horrible idea,” Levenson said. “There’s not enough oversight as it is now. That would remove almost all oversight.”

The former AR on the MAX who provided details of the engineer’s removal said he spoke to The Seattle Times because he hopes for action to reverse the industry’s direction.

He said the two crashes that claimed so many lives in Indonesia and Ethiopia starkly emphasize the need to force the FAA to go back to a DER-style structure, where those working at Boeing on behalf of the FAA are directly overseen by agency technical experts.

“Unfortunately, in our industry, the pendulum swings when people die,” he said. “Let those people’s deaths mean something.”

Seattle Times researcher Miyoko Wolf contributed to this story.

By Dominic Gates and Mike Baker

Early in the development of the 737 MAX, engineers gathered at Boeing’s transonic wind tunnel in Seattle to test the jet’s aerodynamics using a scale model with a wingspan comparable to that of an eagle.

The testing in 2012, with air flow approaching the speed of sound, allowed engineers to analyze how the airplane’s aerodynamics would handle a range of extreme maneuvers. When the data came back, according to an engineer involved in the testing, it was clear there was an issue to address.

Engineers observed a tendency for the plane’s nose to pitch upward during a specific extreme maneuver. After other efforts to fix the problem failed, the solution they arrived at was a piece of software — the Maneuvering Characteristics Augmentation System (MCAS) — that would move a powerful control surface at the tail to push the airplane’s nose down.

This is the story, including previously unreported details, of how Boeing developed MCAS, which played a critical role in two airliners nose-diving out of the sky, killing 346 people in Ethiopia and off the coast of Indonesia.

Extensive interviews with people involved with the program, and a review of proprietary documents, show how Boeing originally designed MCAS as a simple solution with a narrow scope, then altered it late in the plane’s development to expand its power and purpose. Still, a safety-analysis led by Boeing concluded there would be little risk in the event of an MCAS failure — in part because of an FAA-approved assumption that pilots would respond to an unexpected activation in a mere three seconds.

The revised design allowed MCAS to trigger on the inputs of a single sensor, instead of two factors considered in the original plan. Boeing engineers considered that lack of redundancy acceptable, according to proprietary information reviewed by The Seattle Times, because they calculated the probability of a “hazardous” MCAS malfunction to be virtually inconceivable.

As Boeing and the FAA advanced the 737 MAX toward production, they limited the scrutiny and testing of the MCAS design. Then they agreed not to inform pilots about MCAS in manuals, even though Boeing’s safety analysis expected pilots to be the primary backstop in the event the system went haywire.

In the wake of the two crashes, despite an outcry from the public and from some pilot and airline industry officials, Boeing has defended the processes behind its MCAS design decisions and refused to accept blame.

The grounding of the MAX has entered its 15th week. Safety officials around the world are scrutinizing the changes to MCAS that Boeing has proposed to ensure such accidents won’t happen again. And they are assessing what training pilots may need on the new system.

“Safety is our top priority,” Boeing said in a statement. “Through the work we are doing now in partnership with our customers and regulators to certify and implement the software update, the 737 MAX will be one of the safest airplanes ever to fly.”

This investigation examines what’s known about the origins and operation of MCAS ahead of the final official accident-investigation reports, expected late this year for Lion Air Flight 610 and next year for Ethiopian Airlines Flight 302.

Wind-tunnel and simulator tests

Though Boeing was locked into a plan to revamp its popular 737 model, the Seattle wind-tunnel tests in 2012 revealed a problem.

During flight tests to certify an airplane, pilots must safely fly an extreme maneuver, a banked spiral called a wind-up turn that brings the plane through a stall. While passengers would likely never experience the maneuver on a normal commercial flight, it could occur if pilots for some reason needed to execute a steep banking turn.

Engineers determined that on the MAX, the force the pilots feel in the control column as they execute this maneuver would not smoothly and continuously increase. Pilots who pull back forcefully on the column — sometimes called the stick — might suddenly feel a slackening of resistance. An FAA rule requires that the plane handle with smoothly changing stick forces.

The lack of smooth feel was caused by the jet’s tendency to pitch up, influenced by shock waves that form over the wing at high speeds and the extra lift surface provided by the pods around the MAX’s engines, which are bigger and farther forward on the wing than on previous 737s.

This was verified in early simulator modeling, with planes tested in scenarios at about 20,000 feet of altitude, according to one of the workers involved.

While the problem was narrow in scope, it proved difficult to cope with. The engineers first tried tweaking the plane’s aerodynamic shape, according to two workers familiar with the testing. They placed vortex generators — small metal vanes on the wings — to help modify the flow of air, trying them in different locations, in different quantities and at different angles. They also explored altering the shape of the wing.

Two people familiar with the discussions said 737 MAX chief test pilot Ray Craig preferred such a physical solution to solve the plane’s aerodynamics. Philosophically, Boeing had long opposed efforts to create automated actions such as a stick-pusher — a device used on some aircraft that without pilot action pushes the control column forward to lower the jet’s nose — that would seize control of a situation from the pilot, according to one of the people.

But the aerodynamic solutions didn’t produce enough effect, the two people said, and so the engineers turned to MCAS.

It was simple in concept but powerful in effect, quickly solving the issue.

In the midst of a wind-up turn, the software would automatically swivel up the leading edge of the plane’s entire horizontal tail, known as the horizontal stabilizer, so that the air flow would push the tail up and correspondingly push the nose down.

As the pilot pulled on the control column, this uncommanded movement in the background would counter the jet’s tendency to pitch up and smooth out the feel of the column throughout the maneuver.

An engineer recalled Craig testing MCAS for the first time in the simulator.

“Yeah! This is great,” Craig gushed after seeing how MCAS responded, according to the engineer. (Craig left Boeing before the operation of MCAS was revised.)

This original version of MCAS, according to two people familiar with the details, was activated only if two distinct sensors indicated such an extreme maneuver: a high angle of attack and a high G-force.

Angle of attack is the angle between the wing and the oncoming air flow. G-force is the plane’s acceleration in the vertical direction.

How much MCAS moved the tail when activated was a function of the angle of attack and the jet’s speed, said one of the people familiar with the MCAS design who, like many of the sources in this story, asked for anonymity because of the sensitivity of ongoing investigations.

The fix didn’t stir much controversy.

Another Boeing plane, the KC-46 Air Force tanker, has a software-driven system that similarly moves the stabilizer in a wind-up turn and even has the same MCAS name, though the design is very different.

Boeing’s failure analysis

When Boeing was ready to certify the 737 MAX, it laid out its plan for MCAS in documents for the FAA.

Under the proposal, MCAS would trigger in narrow circumstances. It was designed “to address potentially unacceptable nose-up pitching moment at high angles of attack at high airspeeds,” Boeing told the FAA in a proprietary System Safety Assessment reviewed by The Times.

In a separate presentation made for foreign safety regulators that was reviewed by The Times, Boeing described MCAS as providing “a nose down command to oppose the pitch up. Command is limited to 0.6 degrees from trimmed position.”

Two people involved in the initial design plans for MCAS said the goal was to limit the system’s effect, giving it as little authority as possible. That 0.6-degree limit was embedded in the company’s system safety review for the FAA.

The Boeing submission also included an analysis that calculated the effect of possible MCAS failures, with each scenario characterized as either a minor, a major or a hazardous failure — increasingly severe categories that determine how much redundancy must be built in to prevent the event.

Virtually all equipment on any commercial airplane, including the various sensors, is reliable enough to meet the “major failure” requirement, which is that the probability of a failure must be less than 1 in 100,000.

A “major failure” is not expected to produce any serious injuries and is defined more as something that would increase the cockpit crew’s workload. Such systems are therefore typically allowed to rely on a single input sensor.

Boeing analyzed what would happen if, in normal flight mode, MCAS triggered inadvertently up to its maximum authority and moved the horizontal stabilizer the maximum 0.6 degrees.

It also calculated what would happen on a normal flight if somehow the system kept running for three seconds at its standard rate of 0.27 degrees per second, producing 0.81 degrees of movement, thus exceeding the supposed maximum authority.

Why three seconds? That’s the period of time that FAA guidance says it should take a pilot to recognize what’s happening and begin to counter it.

Boeing assessed both of these failure modes as “major.” Finally, the analysis looked at the inadvertent operation of MCAS during a wind-up turn, which was assessed as “hazardous,” defined in a cold actuarial analysis as an event causing serious or fatal injuries to a small number of people, but short of losing the plane (that’s called “catastrophic”).

Hazardous events typically demand more than one sensor — except when they are outside normal flight conditions and unlikely to be encountered, such as a wind-up turn.

According to a document reviewed by The Seattle Times, Boeing’s safety analysis calculated this hazardous MCAS failure to be almost inconceivable: Given the improbability of an airliner experiencing a wind-up turn, compounded by the unlikelihood of MCAS failing while it happened, Boeing came up with a probability for this failure of about once every 223 trillion hours of flight. In its first year in service, the MAX fleet logged 118,000 flight hours.

So even though this original version of MCAS required two factors — angle of attack and G-force — to activate, Boeing’s analysis indicated that just one sensor would be acceptable in all circumstances.

In flight test, MCAS changes

About a third of the way through flight testing in 2016, as first reported by The Seattle Times in March, Boeing made substantial changes to MCAS.

The flight-test pilots had found another problem: The same lack of smooth stick forces was also occurring in certain low-speed flight conditions. To cover that issue too, engineers decided to expand the scope and power of MCAS.

Because at low speed a control surface must be deflected more to have the same effect, engineers increased the power of the system at low speed from 0.6 degrees of stabilizer nose-down deflection to 2.5 degrees each time it was activated.

On the stabilizer, maximum nose down is about 4.7 degrees away from level flight. So with the new increased authority to move the stabilizer, just a couple of iterations of the system could push it to that maximum.

Because there are no excessive G-forces at low speed, the engineers removed the G-force factor as a trigger. But that meant MCAS was now activated by a single angle-of-attack sensor.

One of the people familiar with MCAS’s evolution said the system designers didn’t see any need to add an additional sensor or redundancy because the hazard assessment had determined that an MCAS failure in normal flight would only qualify in the “major” category for which the single sensor is the norm.

“It wasn’t like it was there to cover some safety or certification requirement,” the person said. “The trigger isn’t a safeguard. It tells (the system) when to operate.”

While the changes were dramatic, Boeing did not submit documentation of the revised system safety assessment to the FAA.

An FAA spokesman said the safety agency did not require a new system safety analysis because it wasn’t deemed to be critical.

“The change to MCAS didn’t trigger an additional safety assessment because it did not affect the most critical phase of flight, considered to be higher cruise speeds,” he said.

The person familiar with the details of MCAS’ evolution said Boeing did the extra analysis of the new low-speed, higher-authority changes. He said the effect of the potential failures at low speed was less, and so didn’t add any risk to the prior analysis. So the documents sent to the FAA with the failure analysis were not revised.

“You turn in the answer,” he said. “You don’t have to document all your work.”

MCAS as it was actually implemented differed in another way from what was described in the safety analysis turned in to the FAA.

The failure analysis didn’t appear to consider the possibility that MCAS could trigger repeatedly, as it did on both accident flights. Moving multiple times in 0.6 or 2.5 increments depending on the speed, it effectively had unlimited authority if pilots did not intervene.

Discussions around this new MCAS design appear to have been limited during flight testing.

Two former Boeing test pilots described a culture of pressure inside the company to limit flight testing, which can delay projects at a time when orders are stacking up, costing the company money.

Matt Menza, a different pilot who did test flights on the MAX, recalled times when test pilots at Boeing would have the chance to thoroughly examine systems in what he called a “system-safety murder board” to explore all the potential failures. But he reported that the general corps of test pilots didn’t have a lot of technical details about the MCAS design, such as the single-sensor input.

Boeing never flight-tested a scenario in which a broken angle-of-attack sensor triggered MCAS on its own, instead relying on simulator analysis, according to a person familiar with the process. One of the former test pilots expressed bewilderment that the angle-of-attack failure was never explored in the air.

A variety of employees have described internal pressures to advance the MAX to completion, as Boeing hurried to catch up with the hot-selling A320 from rival Airbus.

Mark Rabin, an engineer who did flight-testing work unrelated to the flight controls, said there was always talk about how delays of even one day can cost substantial amounts. Meanwhile, staff were expected to stay in line, Rabin said.

“It was all about loyalty,” Rabin said. “I had a manager tell me, ‘Don’t rock the boat. You don’t want to be upsetting executives.’”

Do pilots need more training?

Boeing’s system safety analysis of MCAS, in working out the failure probabilities, assumes that the pilots will take steps in response to anything that arises, and will do so quickly.

The pilots’ struggles to control their planes before both MAX crashes suggest that the FAA’s three-second guidance for expected pilot response time, upon which part of Boeing’s system safety analysis was based, needs to be carefully reassessed.

“If the three seconds is not an appropriate amount of time to be able to catch a runaway stabilizer, and it actually takes seven seconds, then … we need to understand that,” said the person familiar with the details of MCAS.

When MCAS is activated in the cockpit and moves the horizontal stabilizer, a large wheel beside each pilot that’s mechanically connected to the stabilizer begins to spin. This is the manual trim wheel. As a last resort to stop a stabilizer moving uncommanded, a pilot can grab and hold the wheel.

The person familiar with MCAS said the wheel will spin noisily and fast, 30 or 40 times, for each activation. Meanwhile the stabilizer movement will increase the force needed to hold the control column, by about 40 to 50 pounds for a 2.5 degree movement. Such uncommanded movement that won’t stop is referred to as a “runaway stabilizer.”

Boeing has said that to deal with this, pilots need first to have basic hand-flying skills — pull the nose up to where you want it, then use the thumb switches on the yoke that connect electrically to the stabilizer to neutralize the forces — and then shut off MCAS with a pilot checklist procedure on how to handle a “runaway stabilizer.”

However on both accident flights, the angle-of-attack sensor failure set off multiple alerts causing distraction and confusion from the moment of takeoff, even before MCAS kicked in.

On the Ethiopian Airlines flight, for example, a “stick shaker” noisily vibrated the pilot’s control column throughout the flight, warning the plane was in danger of a stall, which it wasn’t; a computerized voice repeating a loud “Don’t sink!” warned that the jet was too close to the ground; a “clacker” making a very loud clicking sound signaled the jet was going too fast; and multiple warning lights told the crew that the speed, altitude and other readings on their instruments were unreliable.

Exactly what pilot training for MCAS is appropriate has become a big issue that threatens to prolong the grounding of the MAX.

While the FAA and U.S. airlines seem ready to clear the plane to fly with just iPad training for American pilots on the MCAS fixes, some foreign regulators want more intensive simulator training for all pilots on how to handle a runaway stabilizer.

Early in the process of selling the MAX, according to two people familiar with the discussions, Boeing promised to give Southwest Airlines a substantial rebate for every plane if the MAX required simulator training.

One former MAX worker, Rick Ludtke, said the rebate reported to him by managers was $1 million per plane, a figure another Boeing employee indicated is roughly accurate.

A Southwest spokesperson said, “We do not discuss publicly the specific details of our contractual agreements,” but added that “the purchase of an aircraft is a significant investment, and guarantees for various items … are incorporated into every 737 contract.”

Ludtke and two other former workers described internal pressures during the MAX certification to avoid any changes to the design of the plane that might cause the FAA to lean toward a simulator mandate.

It became a significant point of attention for Michael Teal, the 737 MAX program manager, and Keith Leverkuhn, vice president and general manager of the 737 MAX program, according to a person involved in the discussions. They felt confident based on past experience that the MAX would be approved without simulator training, but they were wary, according to the worker. 

Meanwhile, Boeing’s chief technical pilot on the MAX, Mark Forkner, was also facing pressure, according to another person involved in the project. The person recalled Forkner as frequently anxious about the deadlines and pressures faced in the program, going to some of his peers in the piloting world for help.

As first reported by The New York Times, Forkner suggested to the FAA that MCAS not be included in the pilot manual, according to a person familiar with the discussions.

“Mark never dreamed anything like this could happen,” said Forkner’s attorney, David Gerger. “He put safety first – at this job and in the Air Force.”

U.S. pilot unions have expressed concern at the omission of MCAS from the manual. One reason is that when MCAS activates, it changes somewhat the response of the airplane.

For example, there is a cutout switch in the control column so that when a pilot pulls or pushes in the opposite direction to a runaway stabilizer, it cuts electric power to the stabilizer. When MCAS is active, this cutout switch doesn’t work, which could surprise a pilot who didn’t know about the system.

Boeing ultimately won the FAA’s approval to give pilots just an hour of training through an iPad about the differences between the MAX and the previous 737 generation. MCAS was not mentioned.

The FAA, after internal deliberations, also agreed to keep MCAS out of the manual, reasoning that MCAS was a software code that operates in the background as part of the flight-control system, according to an official familiar with the discussions.

A single sensor

Boeing has avoided accepting direct blame in public, saying MCAS was only one link in a chain of events. Its leaders have also said  MCAS was designed according to the standard procedures it has used for years.

“The 737 MAX was certified in accordance with the identical FAA requirements and processes that have governed certification of previous new airplanes and derivatives. The FAA considered the final configuration and operating parameters of MCAS during MAX certification, and concluded that it met all certification and regulatory requirements,” Boeing said in a statement.

The most controversial detail of the MCAS design has been the reliance on a single angle-of-attack sensor. On both of the deadly flights, everything started with a faulty sensor. In the second crash in Ethiopia, the data trace strongly suggests that the sensor was destroyed in an instant, likely by a bird strike.

There are two such sensors, one on either side of the fuselage. Why didn’t Boeing, especially after discarding the G-force as a trigger, use both angle-of-attack sensors?

The thinking was that requiring input from two angle-of-attack sensors would mean that if either one failed the system would not function.

That has implications not only for safety but for airline costs. If the system is down, a pilot might fly into a situation where it’s needed and find it unavailable. Or the airline might have to take the plane out of service and lose money.

Both factors point toward a principle of not adding complexity: Keep a system as simple as possible.

“You don’t want to disrupt your customer’s operations,” said the person familiar with the MCAS details. And you don’t want to “increase the risk that the system fails when you need it.”

In this case, as simple as possible meant as minimal as the safety regulations allow. Since Boeing’s system safety analysis concluded that one sensor was acceptable, that’s what it went with.

But that’s not the logic followed for a system on the KC-46 Air Force tanker, also called MCAS.

Boeing says the MCAS systems on the MAX and on the tanker share only a name and a similar function, and have completely different avionics.

But they both move the horizontal stabilizer to smooth the pilot stick forces in a wind-up turn. Their basic design architecture can be compared to some extent.

Air Force spokeswoman Ann Stefanek says “MCAS on the KC-46 has two sensors and the system compares the two readings.”

Boeing’s proposed update to MCAS for the MAX will have the same.

Last Sunday at the Paris Air Show, Boeing CEO Dennis Muilenburg reiterated the company’s position that while the original MCAS was properly designed, “we know we can improve it.”

The fixes include relying on two sensors rather than one, limiting MCAS to one rather than multiple activations, and revising the software.

“We are confident that they will result in a safe airplane, one of the safest airplanes ever to fly, and that MCAS will not contribute to a future accident,” he said.

By Dominic Gates, Steve Miletich  and Lewis Kamb 

Seven weeks after the second fatal crash of a 737 MAX in March, a Boeing engineer submitted a scathing internal ethics complaint alleging that management — determined to keep down costs for airline customers — had blocked significant safety improvements during the jet’s development.

The ethics charge, filed by 33-year-old engineer Curtis Ewbank, whose job involved studying past crashes and using that information to make new planes safer, describes how around 2014 his group presented to managers and senior executives a proposal to add various safety upgrades to the MAX.

The complaint, a copy of which was reviewed by The Seattle Times, suggests that one of the proposed systems could have potentially prevented the crashes in Indonesia and Ethiopia that killed 346 people. Three of Ewbank’s former colleagues interviewed for this story concurred.

The details revealed in the ethics complaint raise new questions about the culture at Boeing and whether the long-held imperative that safety must be the overarching priority was compromised on the MAX by business considerations and management’s focus on schedule and cost.

Managers twice rejected adding the new system on the basis of “cost and potential (pilot) training impact,” the complaint states. It was then raised a third time in a meeting with 737 MAX chief project engineer, Michael Teal, who cited the same objections as he killed the proposal.

A version of the proposed system, called synthetic airspeed, was already installed on the 787 Dreamliner.

It was not directly related to the flight-control system — the Maneuvering Characteristics Augmentation System (MCAS) — that contributed to both crashes. But it would have detected the false angle of attack signal that initiated events in both accidents, and so potentially could have stopped MCAS from activating and repeatedly pushing down the nose of each jet.

But installing it in the MAX would likely have meant 737 pilots needed extra training in flight simulators. Running thousands of pilots through simulator sessions would have delayed the jet’s entry into service and added substantial costs for Boeing’s airline customers, damagi

Ewbank’s complaint goes further than the decision not to install this one new system. He describes management as “more concerned with cost and schedule than safety and quality.” And he alleges that in one instance Boeing hid inflight safety incident data from the European Union Aviation Safety Agency (EASA).

As first reported in The Seattle Times, Boeing did an inadequate system safety assessment that missed flaws in the design of MCAS that were central to the two MAX disasters. And Boeing engineers were under pressure to limit safety testing to certify the MAX. These fresh allegations from inside Boeing indicate that the problems with jetmaker’s safety culture may go deeper than MCAS.

Submitted via Boeing’s internal whistleblower system, Ewbank’s complaint alleges that MAX program managers, concerned with avoiding higher costs and more pilot training, were intent on “shutting down trade studies that attempted to modernize the airplane and avoiding awareness of known issues encountered in historical 737 operation.”

Federal investigators

The FBI has interviewed at least two Boeing employees about the complaint. It’s unclear how the Boeing document reached the agency, but federal investigators are known to have issued subpoenas to the company.

Department of Justice prosecutors, Department of Transportation inspectors and Securities and Exchange Commission (SEC) officials are all involved in a wide-ranging federal investigation into possible wrongdoing at Boeing during certification of the MAX that was already under way before the engineer filed his internal complaint in April.

Boeing declined to comment on the details of the ethics complaint. Teal, 737 MAX chief project engineer, could not be reached for comment. The Department of Justice also declined to comment. The Seattle Times is not naming the employees who have been questioned by the FBI to protect the identity of the source of that information.

Ewbank declined to be interviewed. The Seattle Times is naming him because he identified himself in his complaint to Boeing.

The MAX has been grounded worldwide for almost seven months as Boeing works on a comprehensive fix to its flight-control systems that will satisfy air safety regulators around the globe. The final updates to the systems are expected to be submitted to the Federal Aviation Administration (FAA) this month, and Boeing anticipates clearance to return the jet to the sky in November.

Meanwhile, multiple investigations and reviews, internal and external, are examining what caused the deadly crashes. Last week, Boeing’s board announced a revamp of the company’s reporting structures aimed at producing better internal safety oversight. On Monday, Boeing chairman and chief executive Dennis Muilenburg said he’s “taking immediate steps” to implement those recommendations.

The engineer

Ewbank’s ethics complaint expressed concern about the possible personal consequences of stepping forward inside the company.

“Given the nature of this complaint, the fear of retaliation is high, despite all official assurances that this should not be the case,” he wrote. “There is a suppressive cultural attitude towards criticism of corporate policy — especially if that criticism comes as a result of fatal accidents.”

Ewbank wrote that co-workers told him in private they are afraid to speak up about similar safety concerns out of “fear for their jobs.”

In a statement responding to requests for comment this week, Boeing said it “has rigorous processes in place, both to ensure that such complaints receive thorough consideration and to protect the confidentiality of employees who make them.”

“Accordingly, Boeing does not comment on the substance or existence of such internal complaints,” the statement added.

Ewbank’s LinkedIn profile shows he graduated from Embry-Riddle Aeronautical University in 2008 with a degree in aeronautical engineering, then got a master’s at Purdue. After college, he took a job as rocket scientist, doing launch site design engineering at the Kennedy Space Center in Florida with United Space Alliance, the joint venture between Boeing and Lockheed Martin.

He was hired by Boeing in 2010 to work on designing commercial airplane flight deck systems, including the MAX. He now works on airplane systems integration for the 777X program.

However, dissatisfied with his experience on the MAX program, he took a break from Boeing. LinkedIn shows he left the company in April 2015 and returned to work on the 777X only last November.  The reason for the career break is cited in the ethics complaint: his feeling that Boeing management was “squeezing the engineering budget for new programs  …  more concerned with cost and schedule than safety and quality.”

In his first stint at Boeing, he worked on the safety of flight deck systems across multiple jet programs. It put him at the center of what has become one focus of the investigations into the crashes: The systems that tell pilots how their plane is performing in flight and alert them to anything going wrong.

Ewbank’s complaint says his job included “designing appropriate crew alerting and crew procedures based on expected (system) failures.”

Last week, a National Transportation Safety Board (NTSB) report called for improvements to such systems and criticized Boeing’s testing of the MAX for failing to simulate the possible barrage of system failures and warnings the pilots on the crashed flights faced.

The memo

The proposal for system upgrades that Ewbank discusses in his complaint emerged from work he did alongside several veteran employees in Boeing’s Aviation Safety department “to analyze Loss of Control inflight accidents and design flight deck features that would work to break the accident chain of events.”

One was Associate Technical Fellow Randy Mumaw, a cognitive psychologist and “human factors” expert in how pilots react to an airplane’s instruments. Mumaw, who left Boeing in 2015, said that as a non-engineer he can’t assess the technicalities of the synthetic airspeed system. But he said he knew Ewbank as “highly respected and bright.”

The Seattle Times interviewed four former Boeing employees who were involved in the work of assessing the proposed safety upgrades.

Rick Ludtke, a former flight deck integration engineer, worked alongside Ewbank and was a key participant in the proposal, which was presented in an engineering memo titled “Boeing Commercial Airplanes Strategy for Reducing the Risk of Loss of Control Events.”

Ludtke said the purpose of the memo, which Ewbank cites in his complaint, was to “capture the approval” of executives and to try to get a list of six system improvements accepted across Boeing’s airplane programs, including the MAX, which was then in early development.

The memo, which was signed off by Todd Zarfos, the Boeing vice president who heads the company’s engineering design centers, recommended that synthetic airspeed be installed on the MAX “with the next appropriate software update.”

Another veteran Boeing engineer and associate technical fellow, Carlo Ruelos, was the early champion of synthetic airspeed at Boeing.

A pilot flying any airplane needs to know the current airspeed — the plane’s speed relative to the air. Depending on the direction of the wind, that can be faster or slower than the groundspeed, the plane’s speed relative to the earth. Too high an airspeed could stress the airframe. Too low an airspeed could stall the plane.

This key piece of data is measured by pitot-static air pressure sensors, little tubes that stick out of the fuselage on both sides under the cockpit. It’s entered into multiple calculations performed by the flight control computer, so an accurate value is important.

Synthetic airspeed is a new system that provides an additional, indirect calculation of airspeed using different sensors, including the plane’s angle-of-attack sensors. The system enters the airplane’s angle of attack, its weight, the position of its control surfaces and other parameters into a proprietary Boeing algorithm to come up with an independently measured airspeed reading.

The independence of the synthetic reading means that if it matches the direct airspeed readings, it verifies the data as highly reliable. If there’s a discrepancy, the air data is rejected and the plane’s automated systems won’t use it.

Ewbank’s complaint cites a study that found air data reliability, and airspeed awareness in particular, as a “dominant theme” in airplane accidents where the pilots lost control.

The only Boeing airplane using synthetic airspeed today is its latest all-new jet, the 787 Dreamliner.

On the MAX, Ruelos saw an opportunity because the jet had a new integrated air data system box installed that had more computational power than that on the previous 737 NG model. That extra capability, Ruelos decided, would make it possible to add a variant of the 787 synthetic airspeed system to the MAX. And if it could be added, he felt it should be — because it would broadly enhance the reliability of the 737’s air data systems.

Ruelos, now 75 and retired, said in an interview that the pitot and static probes used for standard airspeed measurement“stick out of the airplane and can be damaged by a bird strike. Or something can plug the very small hole.”

So, he said, “I firmly believe that as another means of verifying the air data, (synthetic airspeed) is a key element in maintaining the safety of the airplane.”

“We pushed very hard for it, because safety is always the No.1 priority,” he added. With the new air data avionics box on the MAX, he believed the system was “ready to go” on the new jet.

The crashes

At the time of this proposal, no one had identified MCAS as a concern. Back then, the original design of MCAS was more benign than the final version that went haywire on the two crash flights. It required two sensors to activate — a high angle of attack and and a high G-force —and was less extensive in its ability to push the nose down.

It wasn’t until March 2016 that the MCAS design was changed to depend solely on a single angle-of-attack sensor.

Synthetic airspeed gains significance in the aftermath of the accidents because the system’s cross-check of the independent airspeed readings would raise a red flag if there’s any angle-of-attack sensor fault. If the readings disagree, Ewbank wrote in his complaint, the system as implemented on the 787 is designed to “monitor and detect erroneous angle-of-attack data, and then work to prevent the use of erroneous data by downstream systems.”

While Ewbank prefaces this statement with a careful qualifier —  “It is not possible to say for certain that any actual implementation of synthetic airspeed on the 737 MAX would have prevented the accidents” — his implication is clear: Synthetic airspeed might have stopped MCAS from activating in the circumstances of the two crashed flights.

Ludtke and Ruelos agreed.

There’s “a very good chance” that if Boeing had implemented synthetic airspeed on the MAX, it would have prevented the crashes, Ludtke said.

“In our department, we never designed anything without comparators,” meaning monitors that compare independent sensor readings and de-activate the system if they disagree, he said. “Curtis, I know, had several types of comparators in that synthetic airspeed system.”

Asked separately if synthetic airspeed might have prevented the crashes, Ruelos responded: “I think so. The left and right systems do cross checks, and if there is a discrepancy, it won’t let the automatic system take control of the airplane. … It would disengage and the downstream systems wouldn’t use the data.”

The cost concerns

Of course, Boeing could have achieved the same result in simpler ways, for example if MCAS had been designed from the start to compare readings from the two angle-of-attack sensors instead of only one. Still, in hindsight the rejection of synthetic airspeed seems fateful.

In his complaint, Ewbank puts it down to “a corporate culture … of expediency of design-to-market and cost-cutting.”

“The 737 MAX was designed via piecemeal updates to prevent triggering expensive certification and (pilot) training,” his complaint states.

Ludtke agreed. Synthetic airspeed was rejected “probably because of cost,” he said. He said Boeing had promised the airlines that the MAX would be so minimally different from the prior 737 model that no additional pilot certification or flight simulator training would be necessary.

He said his manager told him Boeing promised MAX launch customer Southwest “$1 million per tail” if the FAA were to require expensive simulator training.

“The MAX program leaders had always mandated that, if it’s not required for function or certification, it’s not going on the airplane,” Ludtke said. They looked upon synthetic airspeed as “a good improvement, but just an improvement,” not a necessity.

“We still tried. Because we believed these aircraft need improving for the quality of pilots we are experiencing,” Ludtke added. “In the old days, before the MAX, that’s how we did business. At the launch of a new program, its leaders would be very interested in including all the latest ideas and safety improvements.

“The MAX was different from the very beginning,” he said. “We’re just going to put these new engines on and the minimum change to make that happen. That’s it. We’re not spending money.”

“That concept broke the company,” Ludtke concluded.

Another former Boeing employee, a veteran test pilot also involved in the assessment of the proposed system changes, wasn’t close enough to the technical details of synthetic airspeed to be sure it would have prevented the accidents, yet agreed that any similar system based on angle of attack likely would have cut out MCAS.

“That’s how you would hope the system would work,” said the pilot, who asked for anonymity to preserve relationships at Boeing.

And the pilot agreed with Ludtke that preserving the MAX’s common type rating — certifying it as just a variant of the prior 737 NG model, rather than a new airplane — and ensuring that airline pilots wouldn’t be required to train for the MAX on flight simulators was “such a huge deal” that it blocked potential updates to the avionics systems.

“I couldn’t believe they kept stretching the 737, both literally (with a longer fuselage) and also in terms of cockpit design,” the pilot said.

The culture

Ray Craig, former chief pilot on the 737 MAX until he retired in 2015, had a very different take. He said he worked with Ewbank and knew him as a “very sharp, very dedicated” engineer.

Yet he defended the safety culture at Boeing and around the MAX program.

“Safety was paramount. If there was something we thought was a safety issue, there was no question, it was taken care of,” Craig said. “But it’s not always a black-and-white decision.”

Lacking full technical details, he wouldn’t venture an opinion about whether synthetic airspeed could have prevented the crashes. “I don’t remember it as ready to go. It wasn’t just a simple plug-and-play,” Craig said. “It wasn’t as program-ready as perhaps some of the folks were thinking. But I don’t remember the exact reason it was shot down.”

Ewbank’s ethics complaint is much broader than the failure to install synthetic airspeed. He attacks the company’s culture around aviation safety and questions Craig’s and Boeing’s assertion that safety is always paramount.

He recounts an episode in his department when he says Boeing hid in-flight safety incidents from Europe’s aviation  regulator. This occurred when EASA found five events where 737s experienced a problem with the autothrottle disconnecting on approach and a confusing alert led to an inappropriate pilot response.

EASA asked if Boeing was aware of any other such events and Ewbank was assigned to search the in-service databases. But when he identified five further similar incidents on 737s, his ethics complaint says his manager decided “to not tell EASA about these events” and that instead “we would fix the issue ourselves.”

Ewbank, a relatively young engineer at the start of his career and with less than six years at Boeing over his two employment stints, even goes so far in the complaint as to directly attack CEO Muilenburg.

He cites Muilenburg’s statement on a quarterly earnings teleconference, just four days before Ewbank filed the ethics complaint, denying that the two recent MAX crashes were due to any “technical slip” by Boeing during the jet’s design or certification. Ewbank calls this “a false statement.”

“When CEO Muilenburg and others state that the Max was a safe airplane as designed, they seriously misrepresent what Boeing Engineering has learned about how data and control functions should be treated,” Ewbank wrote.

Seattle Times researcher Miyoko Wolf contributed to this report.

By Dominic Gates, Steve Miletich and Lewis Kamb

In 2014, Boeing convinced the Federal Aviation Administration (FAA) to relax the safety standards for the new 737 MAX related to cockpit alerts that would warn pilots if something went wrong during flight, according to documents reviewed by the Seattle Times.

Seeking an exception, Boeing relied on a special FAA rule to successfully argue that full compliance with the latest federal requirements would be “impractical” for the MAX and would cost too much.

“They went through the process and weren’t required to step up,” said an FAA safety engineer familiar with how the waiver request was handled and who asked for anonymity because he spoke without agency authorization.

Based on lessons learned from past airline accidents, the FAA regulation stipulates precise design details for the warning displays in the cockpit. These are aimed at ensuring that alerts relay clearly to the pilots what’s going on when a malfunction occurs, catch attention so that they won’t be overlooked, and avert any possible confusion.

During the two fatal MAX crashes that killed 346 people, pilots struggled to understand the cascade of warnings in their cockpits. Last week a National Transportation Safety Board (NTSB) report on those crashes highlighted the crucial role that crew alerting systems play when pilots face an in-flight emergency.

The Seattle Times reviewed the relevant parts of the document that Boeing submitted to the FAA to win its exception. They show the federal regulator struck out four separate clauses that would be requirements for any new jet being produced today.

This meant Boeing avoided having to design a complete upgrade of the 737’s aging flight-crew-alerting system.

The underlying design of the 737 was first certified more than five decades ago, and its airframe and systems have been upgraded in an incremental patchwork ever since. Boeing’s submission reveals the cold actuarial calculus by which such exceptions are granted to allow certification of airplanes, such as the MAX, that are derivatives of older, legacy models.

Following the MAX crashes, such rulings are likely to come under tougher scrutiny in the future.

Boeing declined to comment on the details in this story. The FAA said in a statement that the MAX complies with the “applicable” regulations, then listed some of the criteria under which exceptions from full compliance are granted.

Relaxing the rules

Boeing’s argument in the document, which has not been previously reported, rested most basically on the long service history of the 737. At the time the MAX’s exception was granted, that included more than 300 million hours in the air, almost all accumulated on routinely safe flights.

However, Boeing’s analysis also had to deal with the fact that the 737’s record in the previous 10 years included three fatal crashes where crew alerting was a contributing factor: the 2005 Helios Airways crash in Greece that killed 121 people; the 2009 Turkish Airlines crash in Holland with nine fatalities; and the 2008 Aeroflot-Nord crash in Russia, in which 88 died.

Boeing convinced the FAA that it had dealt with the three distinct issues in each of those crashes.

The submission from Boeing then cited an estimate of the cost of full compliance at more than $10 billion.

This staggering sum included not only the direct cost to Boeing of redesigning the airplane systems but also the expense of additional pilot training that new systems would require — costs that would have been borne by Boeing’s airline customers and would have made the MAX a much less attractive airplane to buy.

In April 2014, the FAA accepted Boeing’s argument that for the MAX, the safety benefit of full compliance with the crew-alerting regulations was “not commensurate with the costs necessary to comply.”

A new urgency

Pilots rely on their instruments to tell them how an airplane is performing in flight and to warn of any system malfunctions. The federal regulations are designed to make such alerts as clear and unambiguous as possible about the nature and severity of any malfunctions.

The early investigation reports into the two MAX crashes show the pilots didn’t understand what their instruments were telling them and failed to handle the emergency as they might have.

Though the accidents were initiated by a failed sensor and a flawed Boeing flight-control system, both the capabilities of the pilots and the design of the crew alerting system played a role in the outcome.

Last week’s NTSB report criticized Boeing for failing to account in its testing of the MAX for the overload of warning messages in the cockpit that occurred during the two fatal flights.

One of the current alerting regulations that the MAX is excused from is relevant to such a scenario. It requires that the system must be designed to prevent or suppress erroneous attention-getting alerts that might interfere with the crew’s ability to focus — such as the “stick-shaker” that vibrated the captain’s control column on both the crashed MAX flights.

Because of a faulty angle-of-attack sensor on each flight, the stick-shaker was warning, falsely, that the jet was close to a stall. But having noted it, the pilots couldn’t stop it. The MAX has no way to suppress that alert. The stick-shaker continued throughout both flights, along with multiple other alerts.

On the Ethiopian Airlines flight that crashed in March, the pilots faced a barrage of alerts throughout the six-minute flight. Besides the stick-shaker, they heard repeated loud “DON’T SINK” warnings that the jet was too close to the ground; a “clacker” making a very loud clicking sound to signal the jet was going too fast; and multiple warning lights telling the crew the speed, altitude and other readings on their instruments were unreliable.

Pilots around the world vary greatly in their flying expertise, especially in their ability to handle a plane when automated systems fail. While many U.S. airline pilots previously have flown military planes for the Air Force, that’s not the experience level in most countries. Further, even a good pilot will have a bad day.

So both Boeing and rival Airbus will in future have to pay increasing attention to “human factors,” meaning the way people interpret and respond to systems and what’s happening around them — which in an airplane depends crucially on the crew alerting system.

A person familiar with the details said that the European Union Aviation Safety Agency (EASA), in its ongoing re-evaluation of the MAX following the two crashes, has already expressed concern to both Boeing and the FAA about inadequacies in the jet’s alerting system, including the constant erroneous stick shaker.

Boeing’s state-of-the-art system

Early in the development of the 737 MAX, Boeing considered equipping the flight deck with its state-of-the-art flight-crew alerting system, called EICAS, the Engine-Indicating and Crew-Alerting System.

It provides pilots visual, aural and tactile warnings as well as written messages on the main flight display when anything goes wrong with either the engines or with the airplane systems, and then also recommends the remedial action needed.

EICAS, designed to take account of the latest human factors studies, is a system that integrates all the interactions between the pilots and the machine they are flying.

Boeing introduced EICAS in the early 1980s when the 757 and 767 jets entered service. The improved alert system was one justification for removing the role of flight engineer to allow those airplanes to fly with two-person crews. It’s been upgraded incrementally since and installed on all subsequent Boeing jets.

But alone among Boeing jets, the 737 was never updated with EICAS, though it was considered at least twice before in previous iterations of the airplane.

It was pushed again for the MAX.

An ethics complaint submitted in April by Boeing engineer Curtis Ewbank and reviewed by the Seattle Times says that Mike Carriker — Boeing’s chief pilot for product development, who flew the first flight of the 787 Dreamliner — proposed studying whether to put EICAS on the MAX, saying “it was necessary for the 737 to be a modern airplane.”

Boeing identified the detailed changes both to the airplane systems and to crew procedures that would be needed to install EICAS on the 737 MAX. But ultimately that plan was abandoned because of “the overall cost,” the ethics complaint states.

In a brief phone interview last week, Carriker declined to discuss details but said installing EICAS on the 737 “would be challenging.” And pointing to the older systems on the MAX compared to other planes like the Dreamliner, he added that “there aren’t enough sensors on the 737.”

Having settled on retaining its older cockpit alerting system, Boeing then needed to convince the FAA that the MAX should not have to meet all the latest federal crew alerting requirements, which are closely aligned with the capabilities of the EICAS system.

Making an exception

A document submitted by Boeing to the FAA in 2012 lays out the airplane description and preliminary data needed to plan the certification work for the MAX and includes an “issue paper” devoted to the MAX’s crew alerting systems.

A Boeing request for an official exemption from the regulations would have required a public notice in the Federal Register and an opportunity for interested parties or the general public to comment. Instead, Boeing followed a standard procedure for being granted such a waiver that was not public.

Instead of an “exemption,” Boeing asked for an “exception” granted under a special FAA procedure called the “Changed Product Rule,” which lays out the conditions under which a new, changed version of an older model can be granted exceptions during certification.

An official FAA advisory circular stipulates that exceptions will be granted if the applicant, in this case Boeing, can demonstrate that compliance is “impractical.” The design must come close to meeting safety requirements, and then demonstrate that “full compliance would require a substantial increase in the outlay or expenditure of resources with a very small increase in the level of safety.”

Boeing’s submission to the FAA cites first the flight history of the 737, going back to 1967. It notes that by 2011 the jet had completed 321 million flight hours and 213 million departures. Broken down by model type, the 737 version prior to the MAX, known as the 737 NG, had completed 80 million flight hours and 42 million departures.

Boeing then documented the 737’s safety record during the previous 10 years. Between 2002 and 2011, it identified three fatal accidents where a deficiency in the flight-crew-alerting system had played a role in the tragedy. These were:

• Helios Airways flight 522 in 2005. Flying at 34,000 feet near Athens, Greece, the crew misinterpreted a horn that sounded to warn of a cabin depressurization, interpreting it as a false and irrelevant alert about the plane’s take-off configuration. The horn sounds were identical for these two distinct alerts. The pilots passed out from lack of oxygen and the plane continued flying in a straight line on autopilot, shadowed by a Greek jet fighter impotent to help. All 121 people on board died when the airliner ran out of fuel and crashed.

Following the accident, Boeing installed a light on the 737’s pilot display to distinguish a depressurization from the other alert.

• Turkish Airlines flight 1951 in 2009. On approach into Amsterdam, a single radio altimeter fed an incorrect low altitude reading to the autothrottle, which duly retarded the engines for landing. The pilots, busy with some checklists, failed to notice until too late a visual alert about the airspeed dropping too low. The plane crashed well short of the runway. Nine people, including three Boeing engineers who were on board by chance, were killed.

Following the accident, Boeing added an extra aural alert — a computerized voice warning — for low airspeed.

• Aeroflot-Nord flight 821 in 2008. Flying through clouds at night in central Russia, the pilot lost spatial awareness as the plane banked dangerously left, activating a BANK ANGLE artificial voice alert. Confused, the captain turned the yoke the wrong way, rolling hard left and worsening the bank angle. The jet flipped upside down. All 88 people on board died in the crash.

Following the accident, Boeing designed a new aural alert that announces “Roll Right” or “Roll Left” as appropriate to counter a dangerous bank angle and also shows the right direction via an arrow on the flight display.

Each of those crashes was at least partly attributed to pilot error. Postmortem tests showed the Russian captain may even have been drunk. Yet in each case, the crew-alerting system could have been better, and was made so after the fact.

The FAA safety engineer said that in accidents where the pilots are blamed, “many times you’ll find the indication and alerting system provided confusing or misleading information.”

Boeing argued that the exception for the MAX was justified by the long history of safe 737 flights and the fact that it had addressed the separate alerting issues in each of these fatal accidents.

“There is no reason to believe the future rate of accidents for the 737-8 (MAX) will be significantly different from the 737 NG historical record,” the document states.

The submission to the FAA also points to the “existing common and proven alerting methodology” on the approximately 6,400 Boeing 737s then flying worldwide. It adds that the MAX won’t represent the majority of the world 737 fleet until around 2030, which means airlines would be flying mixed fleets for “two generations of 737 pilots.”

Boeing contended that keeping the MAX systems common with the systems on the prior 737 model would be preferable, to avoid confusion as pilots move between the two types of aircraft.

The FAA in its statement Wednesday listed some of the factors considered in agreeing that an aircraft complies with the rules sufficiently to be certified: “these factors include areas of change (in the airplane design), aircraft service experience and actions taken following earlier accidents.”

There is one glaring omission from that list, a factor that nevertheless the FAA guidelines clearly state will be taken into account: The matter of cost.

Yet Boeing’s argument in the MAX certification document finally arrives at that detail: the cost to Boeing and to its airline customers.

Boeing said a “significant design change” would be required if it had to comply with the complete set of federal crew alerting regulations.

“Compliance would also require revision to the entire system of training and documentation that supports the alerting methodology, as used by 75,000 pilots and a large number of airline mechanics and engineers,” the document states.

Boeing estimated the cost of the design, training and documentation changes to achieve full compliance for the 737 MAX would be “greater than $10 billion” in 2013 dollars.

As a result of the two MAX accidents, Boeing has already racked up more than $8.3 billion in extra costs through July,  including a $5.6 billion write-off last quarter, a $2.7 billion addition to the projected future costs of producing the 737, and a payout of $50 million in initial compensation to the families of victims.

The cost has grown since as the grounding of the MAX fleet goes on, and further compensation costs to the families of victims, to customer airlines and to suppliers will likely continue to mount through next year.

The final bill, not even counting Boeing’s potential loss of orders and future market share, will almost certainly far exceed $10 billion.

Those outlays weren’t anticipated during development of the jet. So Boeing’s submission to the FAA concluded that the $10 billion estimate to achieve compliance met the standard for granting an exception, because the effort in terms of cost and changes to manufacturing “would not be commensurate with a small incremental safety gain.”

The FAA accepted this argument and granted Boeing’s request.

A Boeing engineer, who also asked for anonymity to protect his job, was troubled by the way the company’s analysis discounted the Helios, Turkish and Aeroflot 737 crashes.

“Yes, Boeing went and fixed each problem,” said the engineer in an interview. “It did so only after a fatal accident. They are being reactive. Boeing could have been proactive on the 737.”

He said the MAX was another missed opportunity to be proactive on safety upgrades.

In addition, those fixes Boeing developed after the three crashes are not necessarily installed on all the older 737s now in service globally. The FAA did not mandate two of them — the aural alerts that resulted from the Turkish and Aeroflot accidents — in airworthiness directives that would require airlines to comply.

So although U.S. airlines have voluntarily installed those alerts, there may be overseas airlines flying 737s that have not done so.

The FAA engineer agreed that safety shouldn’t depend on an after-the-fact response to fatal accidents. Still, he wasn’t ready to dismiss Boeing’s overall contention that a full upgrade to such an old design wasn’t practical on the MAX.

“Why force a change that would have a huge impact on the industry with no big increase in safety?” he asked. “It’s not a totally invalid argument.”

“It is old technology,” the engineer added. “The 737 flight deck display system is not anywhere near state of the art. But Boeing contends the pilots know it well.”

Clarification: This story was updated Oct.3 to clarify that the relevant section of the federal crew alerting regulation would have required Boeing to redesign the 737 MAX’s alerting system to either prevent or to provide a means to suppress a constant erroneous stick shaker. The MAX was excused from having to meet this requirement. The previous wording referred only to suppressing the stick shaker rather than “preventing or suppressing.”

By Mike Baker and Dominic Gates 

Boeing has long embraced the power of redundancy to protect its jets and their passengers from a range of potential disruptions, from electrical faults to lightning strikes.

The company typically uses two or even three separate components as fail-safes for crucial tasks to reduce the possibility of a disastrous failure. Its most advanced planes, for instance, have three flight computers that function independently, with each computer containing three different processors manufactured by different companies.

So even some of the people who have worked on Boeing’s new 737 MAX airplane were baffled to learn that the company had designed an automated safety system that abandoned the principles of component redundancy, ultimately entrusting the automated decision-making to just one sensor — a type of sensor that was known to fail. Boeing’s rival, Airbus, has typically depended on three such sensors.

“A single point of failure is an absolute no-no,” said one former Boeing engineer who worked on the MAX, who requested anonymity to speak frankly about the program in an interview with The Seattle Times. “That is just a huge system engineering oversight. To just have missed it, I can’t imagine how.”

Boeing’s design made the flight crew the fail-safe backup to the safety system known as the Maneuvering Characteristics Augmentation System, or MCAS.

The Times has interviewed eight people in recent days who were involved in developing the MAX, which remains grounded around the globe in the wake of two crashes that killed a total of 346 people.

A faulty reading from an angle-of-attack sensor (AOA) — used to assess whether the plane is angled up so much that it is at risk of stalling — is now suspected in the October crash of a 737 MAX in Indonesia, with data suggesting that MCAS pushed the aircraft’s nose toward Earth to avoid a stall that wasn’t happening. Investigators have said another crash in Ethiopia this month has parallels to the first.

Boeing has been working to rejigger its MAX software in recent months, and that includes a plan to have MCAS consider input from both of the plane’s angle-of-attack sensors, according to officials familiar with the new design.

“Our proposed software update incorporates additional limits and safeguards to the system and reduces crew workload,” Boeing said in a statement.

But one problem with two-point redundancies is that if one sensor goes haywire, the plane may not be able to automatically determine which of the two readings is correct, so Boeing has indicated that the MCAS safety system will not function when the sensors record substantial disagreement.

Some observers, including the former Boeing engineer, think the safest option would be for Boeing to have a third sensor to help ferret out an erroneous reading, much like the three-sensor systems on the airplanes at rival Airbus. Adding that option, however, could require a physical retrofit of the MAX.

Andrew Kornecki, a former professor at Embry-Riddle Aeronautical University who has studied redundancy systems in Airbus and Boeing planes, said operating the automated system with one or two sensors would be fine if all the pilots were sufficiently trained in how to assess and handle the plane in the event of a problem. But, he said, if he were designing the system from scratch, he would emphasize the training while also building the plane with three sensors.

“As they say: belt and suspenders,” Kornecki said.

The design

Boeing had been exploring the construction of an all-new airplane earlier this decade. But after American Airlines began discussing orders for a new plane from Airbus in 2011, Boeing abruptly changed course, settling on the faster alternative of modifying its popular 737 into a new MAX model.

Rick Ludtke, a former Boeing engineer who worked on designing the interfaces on the MAX’s flight deck, said managers mandated that any differences from the previous 737 had to be small enough that they wouldn’t trigger the need for pilots to undergo new simulator training.

That left the team working on an old architecture and layers of different design philosophies that had piled on over the years, all to serve an international pilot community that was increasingly expecting automation.

“It’s become such a kludge, that we started to speculate and wonder whether it was safe to do the MAX,” Ludtke said.

Ludtke didn’t work directly on the MCAS, but he worked with those who did. He said that if the group had built the MCAS in a way that would depend on two sensors, and would shut the system off if one fails, he thinks the company would have needed to install an alert in the cockpit to make the pilots aware that the safety system was off.

And if that happens, Ludtke said, the pilots would potentially need training on the new alert and the underlying system. That could mean simulator time, which was off the table.

“The decision path they made with MCAS is probably the wrong one,” Ludtke said. “It shows how the airplane is a bridge too far.”

Boeing said Tuesday that the company’s internal analysis determined that relying on a single source of data was acceptable and in line with industry standards because pilots would have the ability to counteract an erroneous input.

In addition to the imminent software fix for the MCAS, people familiar with Boeing’s plans said the company now intends to make standard two features that previously were optional add-ons at extra cost.

The MAX cockpit will now include a warning light that will illuminate when the two angle-of-attack sensors disagree. And airlines can opt to add, free of charge, angle-of-attack data to the primary flight display.

The company has also started holding information sessions with airlines and regulators about the proposed software fix.

Even before the MCAS system activated during the 737 MAX crash of Lion Air Flight 610 last October, the flight’s data showed signs of a problem.

While the pilots didn’t know it, the plane’s two angle-of-attack sensors were recording substantial disagreements even before takeoff. Once the plane left the ground, the pilots immediately got warnings about the airspeed and the risk of a stall.

The pilots managed to ascend because the MCAS system isn’t designed to operate until pilots retract the flaps used on takeoff. It also doesn’t operate in autopilot mode.

Once the Lion Air pilots retracted the flaps at an altitude of 5,000 feet, however, the MCAS interpreted the erroneous angle-of-attack information and automatically swiveled the airplane’s horizontal tail so as to push the jet’s nose sharply down.

Although the pilot countered the horizontal tail movement and brought the nose back up, the faulty signal continued, so the MCAS pushed it back down again repeatedly during a mortifying 12-minute roller-coaster ride. The pilot lost control and the plane plunged into the sea, killing 189 people.

After the crash this month of another 737 MAX, Ethiopian Airlines Flight 302, initial reports suggest that its shorter flight trajectory was similar. And a part found in the wreckage, the jackscrew, shows that on impact, the horizontal tail was swiveled so as to point the nose sharply down. These clues mean the MCAS is suspected in that tragedy also.

Pilot perspective

Matt Menza, a former Boeing pilot who worked on the MAX, said that during flight testing of planes ready for delivery, he wasn’t aware of any events that indicated a problem with the stall warning or the MCAS system. But he said an ideal system would have been built on two angle-of-attack probes, so that a single bad value wouldn’t cause problems.

Menza and two other pilots who have worked on the MAX said they were unaware that the system used only one AOA probe.

Still, Menza pointed out that handling uncommanded inputs from the MCAS would be the same as past procedures for any similar problems, with pilots able to easily flip cutout switches to regain manual control.

“A properly trained pilot should be able to solve an MCAS anomaly or any uncommanded flight-control input through procedures that are taught to all 737 pilots,” said Menza, noting that the emergency information Boeing distributed in December reiterated those procedures.

Boeing has contended since the Lion Air crash that the pilots, even though they’d been told nothing about the MCAS, should still have realized that the nose was turning down because of uncommanded movement of the horizontal tail. A large wheel beside the pilot is connected to the tail and would have spun each time the horizontal tail moved. 

Boeing told The Times Tuesday that the company’s internal analysis determined that a pilot would be able to counteract an erroneous command by using trim switches on the control column, or by following the standard checklist to use cutoff switches that would have turned off all automatic movement of the horizontal tail.

When the preliminary investigation report into the Lion Air crash was published a month after the accident, Boeing issued a long statement that emphasized this perspective and pointed out that on the day before the crash, a different flight crew on that same jet had encountered similar behavior and had hit the cutoff switches, which allowed the flight to continue uneventfully.

On a flight the day before the Lion Air crash, Bloomberg reported, a third pilot who happened to be on board helped the two pilots figure out that they needed to trip the cut-out switches. So while three heads troubleshooting the problem managed to work out the correct response, it appears that on the subsequent flight, the two pilots were overwhelmed.

Two or three sensors

Peter Seiler, a professor at the University of Minnesota who previously worked on the flight-control electronics for the Boeing 787 aircraft, said it would be highly unusual to have a safety-critical system dependent on a single sensor.

“It’s a huge part of the design. It’s a huge part of the certification process,” Seiler said.

But Seiler said he thinks the MAX would be fine if the MCAS depended on two angle-of-attack sensors. If they disagree with each other substantially, the plane will know that one is malfunctioning and can then prevent the MCAS system from engaging. Seiler said pilots can be made aware.

Since it would be an unusual circumstance, Seiler said he thinks it would be fine for the pilots to continue flying for a few hours without the MCAS safety protection activated before getting the sensor fixed when back on the ground.

“The only issue you then get is if the system failed and the pilot is confused,” Seiler said. “You don’t want to operate the airplane all the time that way.”

The sensors don’t fail often, but FAA records reviewed by The Times show it’s happened on a wide variety of aircraft from Boeing and other manufacturers, including a 2009 flight of a 737 out of Dallas-Fort Worth and a 2013 flight of a 747.

In a 2014 flight of a 767 out of Miami, records show the flight crew reported a disagreement in airspeed readings after takeoff, according to FAA records. An emergency was declared and the plane returned to the airport, and the left angle-of attack sensor was replaced.

Angle-of-attack sensors have been around since the 1940s. On its airplanes, including the 737, Boeing uses two angle-of-attack sensors, one on either side of the plane, with each sensor feeding the angle measurement to instruments on the corresponding side of the cockpit.

If the sensor on one side indicates the plane is at a stall angle, it will trigger warnings for the pilot on that side of the plane, including a “stick shaker.” That pilot’s control column will begin to shake.

If only one side gives such warnings, the cockpit crew has the responsibility of assessing the overall circumstances of the flight to determine which side is correct and proceed accordingly.

In contrast, three angle-of-attack sensors are the norm on airplanes from Airbus, which also uses them to automatically move the plane’s nose down in the event of a stall.

Researchers who have studied Airbus’ system have said it considers all three sensor readings and generally relies on the middle of the three. If one of the sensors drifts far out of range, that sensor is ignored, and the flight-control system continues on using an average of the two remaining sensors.

That triple-sensor system isn’t foolproof, however.

In 2008, on a customer-acceptance flight of an Airbus A320, two of the angle-of-attack sensors froze and those two sensors then outvoted the third. When the pilots went to demonstrate the stall-prevention system, they were not aware of the malfunctioning sensors. The plane crashed, killing the seven people on board.

The same problem arose again on a 2014 Airbus A321 Lufthansa flight leaving Spain. Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. This time, after a drop in altitude, the pilots were able to regain control and complete the flight.

By Dominic Gates

While conducting newly stringent tests on the Boeing 737 MAX flight control system, the Federal Aviation Administration (FAA) in June uncovered a potential flaw that now has spurred Boeing to make a fundamental software-design change.

Boeing is changing the MAX’s automated flight-control system’s software so that it will take input from both flight-control computers at once instead of using only one on each flight. That might seem simple and obvious, but in the architecture that has been in place on the 737 for decades, the automated systems take input from only one computer on a flight, switching to use the other computer on the next flight.

Boeing believes the changes can be accomplished in time to win new regulatory approval for the MAX to fly again by October. Significant slipping of that schedule could lead to a temporary halt in production at its Renton plant where 10,000 workers assemble the 737.

After two deadly crashes of Boeing’s 737 MAX and the ensuing heavy criticism of the FAA for its limited oversight of the jet’s original certification, the agency has been reevaluating and recertifying Boeing’s updated flight-control systems.

It has specifically rejected Boeing’s assumption that the plane’s pilots can be relied upon as the backstop safeguard in scenarios such as the uncommanded movement of the horizontal tail involved in both the Indonesian and Ethiopian crashes. That notion was ruled out by FAA pilots in June when, during testing of the effect of a glitch in the computer hardware, one out of three pilots in a simulation failed to save the aircraft.

The thoroughness of the ongoing review of the MAX flight controls in light of the two crashes is apparent in how a new potential fault with a microprocessor in the flight-control computer was discovered during the June testing. Details of that fault not previously reported were confirmed both by an FAA official and by a person at Boeing familiar with the tests.

In response to finding that new glitch, Boeing developed the plan to fundamentally change the software architecture of the MAX flight-control system and take input simultaneously from the two flight-control computers that are standard on the 737.

“This is a huge deal,” Peter Lemme, a former flight-controls engineer at Boeing and avionics expert, said about the change.

Lemme said the proposed software architecture switch to a “fail-safe,” two-channel system, with each of the computers operating from an independent set of sensors, will not only address the new microprocessor issue but will also make the flawed Maneuvering Characteristics Augmentation System (MCAS) that went haywire on the two crash flights more reliable and safe.

“I’m overjoyed to hear Boeing is doing this,” Lemme said. “It’s absolutely the right thing to do.”

According to a third person familiar with the details, Boeing expects to have this new software architecture ready for testing toward the end of September. Meanwhile, it will continue certification activities in parallel so that it can stick to its announced schedule and hope for clearance from the FAA and other regulators in October.

Flipping bits

When Boeing announced June 26 that a new potential flaw had been discovered on the MAX — this time in a microprocessor in the jet’s flight-control computer — it even caught Boeing CEO Dennis Muilenburg by surprise.

Speaking at a conference in Aspen, Colorado, that morning, Muilenburg reiterated a prior projection that the MAX could be carrying passengers again by “the end of summer.” Later that day, Boeing announced the problem in a Securities and Exchange Commission filing, and soon after projected that the issue could add a further three months’ delay.

What the FAA was testing when it discovered this new vulnerability was esoteric and remote. According to the person familiar with the details, who asked for anonymity because of the sensitivity of the ongoing investigations, the specific fault that showed up has “never happened in 200 million flight hours on this same flight-control computer in [older model] 737 NGs.”

In sessions in a Boeing flight simulator in Seattle, two FAA engineering test pilots, typically ex-military test pilots, and a pilot from the FAA’s Flight Standards Aircraft Evaluation Group (AEG), typically an ex-airline pilot, set up a session to test 33 different scenarios that might be sparked by a rare, random microprocessor fault in the jet’s flight-control computer.

This was standard testing that’s typically done in certifying an airplane, but this time it was deliberately set up to produce specific effects similar to what happened on the Lion Air and Ethiopian flights.

The fault occurs when bits inside the microprocessor are randomly flipped from 0 to 1 or vice versa. This is a known phenomenon that can happen due to cosmic rays striking the circuitry. Electronics inside aircraft are particularly vulnerable to such radiation because they fly at high altitudes and high geographic latitudes where the rays are more intense.

A neutron hitting a cell on a microprocessor can change the cell’s electrical charge, flipping its binary state from 0 to 1 or from 1 to 0. The result is that although the software code is right and the inputs to the computer are correct, the output is corrupted by this one wrong bit.

So for example, a value of 1 on a single bit might indicate that the jet’s wing flaps are up, while a 0 would mean they are down. A value of 1 on a different bit might tell the computer that the MAX’s problematic flight-control system called MCAS is engaged, while a 0 would indicate it is not.

This isn’t as alarming as it may sound. There are standard ways to protect against such bit flips having any dangerous impact on an airplane system, and FAA regulations require that this possibility be accounted for in the design of all critical electronics on board aircraft. The simulator sessions in June were designed to test for any such vulnerability.

During the tests, 33 different scenarios were artificially induced by deliberately flipping five bits on the microprocessor, an error rate determined appropriate by prior analysis. For all five bits, each 1 became a 0 and each 0 became a 1. This is considered a single fault, on the assumption that some cause, whether cosmic rays or something else, might flip all five bits at once.

For these simulations, the five bits flipped were chosen in light of the two deadly crashes to create the worst possible combinations of failures to test if the pilots could cope.

In one scenario, the bits chosen first told the computer that MCAS was engaged when it wasn’t. This had the effect of disabling the cut-off switches inside the pilot-control column, which normally stop any uncommanded movement of the horizontal tail if the pilot pulls in the opposite direction. MCAS cannot work with those cut-off switches active and so the computer, fooled into thinking MCAS was operating, disabled them.

Since MCAS exists only on the MAX, not on earlier 737 models, this potential failure applies only to the MAX.

A second bit was chosen to make the horizontal tail, also known as the stabilizer, swivel upward uncommanded by the pilot, which has the effect of pitching the plane’s nose down. Other bits were flipped to add three more complications.

Even though the MCAS system that pushed the nose down on the two crash flights had not been activated, these changes in essence gave the FAA test pilots in the simulator an emergency situation similar to what transpired on those flights. This was deliberate. The FAA demanded, with knowledge about the crashes, that this scenario be rigorously reexamined in a new System Safety Analysis of the MAX’s flight controls.

“We were deliberately emulating some aspects of MCAS in a theoretical failure mode,” the person familiar with the tests said.

This person emphasized how extremely improbable it is that five single bits on the microprocessor would flip at once and that the random bits would make these specific critical changes to the aircraft’s systems.

“While it’s a theoretical failure mode that has never been known to occur, we cannot prove it can’t happen,” he said. “So we have to account for it in the design.”

He added that early published accounts of the fault suggesting that the microprocessor had been overwhelmed and its data-processing speed slowed, causing the pilot-control column thumb switches that move the stabilizer to respond slowly, were inaccurate.

Lemme said he was happy to learn this because those accounts hadn’t made sense technically. And he said that the description of the fault and the chosen combination of random bit flips represent “a terribly worst-case condition that I cannot imagine happening in reality.”

Dwight Schaeffer, a former senior manager at Boeing Commercial Electronics, the company’s one-time in-house avionics division, agreed. “Five independent bit flips is really an extremely improbable event,” he said.

A crash in the simulator

What happened in the initial simulated run of this fault scenario in June is that the FAA test pilots handled the emergency using the standard procedure for a “runaway stabilizer” and recovered the aircraft. But they felt it took too long and that a less attentive pilot caught by surprise might have had a worse outcome.

FAA guidelines say that if an emergency arises on a plane flying by autopilot, the assumption is that a pilot will begin to respond within 3 seconds. If the plane is being flown manually, the assumption is 1 second.

That may seem a very short response time, but it’s not dissimilar to what a driver would be expected to do if, for example, a car skidded on ice or a tire blew. Still, not every driver and not every pilot is attentive.

“It took too long to recover,” said the FAA official familiar with the tests, who also asked for anonymity because of the sensitivity of ongoing investigations. “An important aspect of these simulations is to capture how a representative airline pilot would respond to the situation.”

So again in light of what happened in the crashes, the FAA pilots took a further step. They flew the same fault scenario again, this time deliberately allowing the fault to run for some time before responding. This time, one of the three pilots didn’t manage to recover and lost the aircraft.

Reclassified as “catastrophic”

In testimony Wednesday before a U.S. Senate Appropriations Subcommittee hearing on FAA oversight, Ali Bahrami, associate FAA administrator for aviation safety, confirmed this.

Describing what was tested in June as “a particular failure that was extremely remote,” Bahrami said “several of our pilots were able to recover. But there was one or so that could not recover successfully.”

According to a second FAA source, it was the AEG pilot, representing a typical U.S. airline captain, who failed to recover the jet.

That outcome changed everything for Boeing.

Prior to that, Boeing had classified this failure mode as a “major fault,” a category that can be mitigated by flight-crew action. The one pilot’s failure to recover immediately changed the classification to “catastrophic,” and FAA regulations require that no single fault can be permitted to lead to a catastrophic outcome. That meant Boeing must fix it and eliminate the possibility.

“There are active means of protecting against bit flips,” said retired Boeing electronics manager Schaeffer. “We always built it into our own software.”

One standard way to fix such a problem is to have the second independent microprocessor inside the same flight-control computer check the output of the first. If the second processor output disagrees with that of the first processor for some specific automated flight control, then no automated action is initiated and the pilot must fly manually.

“Now it takes two processors to fail to get the bad result,” the person familiar with the tests said. “You are no longer in the realm of a single point failure.”

A radical redesign

Boeing could have just rewritten the software governing what functions are monitored within the flight-control computer to eliminate this failure scenario.

Instead, it’s decided to make a much more radical software redesign, one that will not only fix this problem but make the MAX’s entire flight-control system — including MCAS — more reliable, according to three sources.

This change means the flight-control system will take input from both of the airplane’s flight computers and compare their outputs. This goes beyond what Boeing had previously decided to do, which is to adjust the MCAS software so that it took input from two angle of attack sensors instead of one.

The problem with that earlier approach is that if something serious goes wrong with the single flight computer receiving this input — whether it’s the bit flipping issue, or a memory corruption or a chip failure of any kind — then the computer output to the flight controls could be wrong even if both angle of attack sensors are working correctly.

For the MAX, the new MCAS was simply added to an existing 737 flight control system called the Speed Trim System, which was introduced with this one-channel computer architecture on the older model 737-300 in the 1980s.

With the proposed dual-channel configuration, both computers will be used to activate the automated flight controls. They will each take input from a wholly independent set of sensors (air speed, angle of attack, altitude and so on) and compare outputs. If the outputs disagree, indicating a computer fault, the computers will initiate no action and just let the pilot fly manually.

In other words, the new system will detect not only any disagreement between the sensors but also check for any processing error in interpreting the information from the sensors.

“This is a really good solution,” said Lemme, adding that “it should have been designed this way” from the beginning of the flight control system in the 1980s.

This raises the separate question of why the potential microprocessor fault discovered in June wasn’t caught in the original System Safety Analysis when the MAX was certified.

That original System Safety Analysis, as The Seattle Times reported in March, was performed by Boeing, and FAA technical staff felt pressure from managers to sign off on it. And as reported in May, there was also pressure from Boeing managers on the engineers conducting the work to limit safety testing during the analysis.

The person familiar with the testing said the new tests in June were informed by the knowledge of what had happened in the crashes, especially the erroneous activation of MCAS that pushed down the nose of the aircraft on both flights.

“It was a reassessment in light of everything else going on in the world with MCAS,” he said. “It’s a different set of eyes, asking a different set of questions.”

David Hinds, a retired Boeing flight controls and autopilot expert, said that clearly “something got missed” in the original MAX certification of MCAS and now this microprocessor fault.

“I’d like to think you’d catch this on first pass,” said Hinds. “They should have looked harder at some of this.”

By Lewis Kamb

At 8 a.m. a year ago today, Fenlix was driving to a business meeting outside the Indonesian capital of Jakarta when he pulled off for coffee and felt his phone buzz with the text message that changed everything.

An old friend had sent a group text alerting Fenlix and others that Lion Air Flight JT610 — a plane headed to their hometown 70 minutes away — was missing somewhere over the Java Sea. He instantly thought of his big brother, Verian Utama, and his chest tightened.

“I know my brother is taking a flight back to my hometown that morning,” recalled Fenlix, a 31-year-old businessman who, like many Indonesian citizens, goes by only one name. “But I have no idea what is the flight number.”

Fenlix told himself not to panic and dialed his father.

Dad, what’s the number of Verian’s flight, Fenlix asked him.

I don’t remember, his father replied.

Is it JT610?

Yes, yes, that’s it.

Fenlix trembled.

“Oh my god, it’s like my heart is stopping,” he recalled of his reaction. “I’m shaking, I think it’s terrible, something bad has happened. So I started texting my brother just to check. I wrote, ‘Bro?’ But there was nothing, no response. And I realized he’s there, my brother was on that plane.”

Fenlix’s experience of discovering the fate of his only sibling played out again and again last Oct. 29, as relatives and loved ones of the 189 people aboard the Boeing 737 MAX jet learned of its crash and faced their own devastation.

Less than five months later, another  737MAX jet — Ethiopian Airlines Flight 302 — nosedived from the sky in similar fashion, killing all 157 passengers on board and thrusting the world’s largest commercial airplane manufacturer into crisis.

Lawsuits, a criminal probe, congressional inquiries, accident investigations and widespread regulatory scrutiny have followed as the 737 MAX — Boeing’s best-selling aircraft — remains indefinitely grounded.

As the somber anniversary of the Lion Air crash drew nearer, new, troubling details continued to emerge about the ill-fated flights, the MAX’s suspect flight-control system, the Federal Aviation Authority’s certification program and Boeing’s safety culture in general.

But for Fenlix and his family, the past year mostly has been filled with prayer and quiet mourning, he said, as they try to face a new reality of carrying on without a beloved brother, son, husband and father.

Verian Utama, 31, was a successful building contractor and a relatively new husband and father, who also pursued a passion for bicycling as an official retailer of high-end Italian racing bikes.

“Many, many people liked him,” Fenlix said. “We all miss him.”

Grief, complicated by legalities

Often lost to outsiders is the “re-victimization” that families encounter in the aftermath of a plane crash, said Charles Herrmann, a Tacoma-based attorney who represents 44 families of Lion Air crash victims and has handled multiple air-crash cases during his 36-year law career.

“They’re victimized three times,” Herrmann said. “First by the crash, when they lose their loved ones. Next, by the insurance company that wants them to sign off on something that is significantly lower than what a case would get in the courtroom. Finally, they get victimized by lawyers, who start making promises of unrealistic amounts of money they’ll be getting in just a few months. The truth is, it’s not going to be that much money and it’s going to take a long time. They get all these people descending on them after a tragedy.”

Complicating the grieving process is a controversy that emerged shortly after the Lion Air crash over the airline’s handling of mandated insurance payments to heirs of the victims. Before doling out what amounts to about $90,000 per passenger, Lion Air — Indonesia’s largest privately run airline — is requiring heirs to sign a release forfeiting any further liability claims against Lion Air, Boeing and potentially hundreds of other defendants.

Lawyers for the families contend the waiver pushed by the airline and its insurer, Global Aerospace, violates provisions in Indonesian aviation law that forbid carriers from attaching “special requirements” to insurance payments. Herrmann, who noted that Global Aerospace, an international company, also insures Boeing, fired off a scathing letter to Lion Air officials in April, contending the so-called “release and discharge” waiver is illegal, invalid and inhumane.

“In my letter, I scratch my head as to why Lion Air is trying to get releases for Boeing,” Herrmann said. “But it seems Global Aerospace is the puppeteer behind the scenes. They’re taking advantage of these victims.”

Latief Nurbana, an Indonesian government official whose 24-year-old son, Muhammad Luhtfi Nurramdhani, died in the Lion Air crash, said some of the crash victims’ families feared they’d miss out on a needed source of income unless they agreed to the airline’s terms.

“Some of the families signed the R&D (release and discharge document), but we didn’t,” said Nurbana, whose son, a postal manager, left behind a wife seven months’ pregnant with the couple’s first child. “Frankly, Lion (Air) treated us very badly.”

Global Aerospace declined to comment. Lion Air did not respond to multiple requests for comment.

Boeing did not specifically respond to questions about the airline’s victim-compensation methods or its connections to them.

Separately, Boeing has set up its own $50 million assistance fund to compensate the families of victims of both 737 MAX crashes. To allocate the $144,500-per-family payments, the company has assigned outside administrators Camille Biros and Kenneth Feinberg, who have managed similar funds for such high-profile disasters as the 9/11 and Boston Marathon terrorist attacks, and mass shootings in Las Vegas and Orlando.

“I think it’s important to emphasize that this is something Boeing did totally voluntarily and separate from any litigation that might happen,” Biros said last week. “And there are no strings attached; Boeing isn’t requiring anyone to sign a release.”

As of last week, 34 families of victims in both crashes had applied for claims from the fund, with payments made to 15 of them, Biros said. Families have until the end of the year to submit a claim, she said.

Utama’s family has applied for compensation from the Boeing fund, but has refused to sign Lion Air’s waiver, Fenlix said.

A week before the Ethiopian crash, Fenlix and his family opted to join an expanding lawsuit against Boeing spearheaded by Herrmann and his partner, former Pierce County Prosecutor Mark Lindquist.  The suit, which so far includes 24 families, names Fenlix as the personal representative for Utama’s parents, his unemployed widow, Friscilla, and his now-2-year-old son, Willfred.

Fenlix’s case is among about 60 lawsuits so far filed against the aerospace company on behalf of more than 150 families or estates of Lion Air crash victims. Attorneys have argued that because the MAX jets were built at Boeing’s factory in Renton, the cases should be litigated in the United States.

The suits are pending before a federal judge in Chicago, where Boeing is headquartered. The judge has ordered mediation, but should those negotiations fail, Boeing may try to get the cases moved to Indonesia, where damage judgments tend to be significantly smaller. Herrmann’s firm already has settled four cases with Boeing over the Lion Air crash.

For Fenlix and his family, the lawsuit isn’t motivated solely by money.

“What we really want for the lawsuit is to tell them that when they design these airplanes, they need to be careful; they need to really think about the safety,” said Fenlix, who runs a company that builds and supplies industrial manufacturing equipment.

“We don’t care how much money they want to give. We want the person back. We want my brother. But they cannot bring him back to life, right? So then, we don’t want this to happen to any other families.”

“He was a good guy”

On the morning of the crash, Utama and his friend and business partner, Andrea Manfredi, boarded the 70-minute flight from Jakarta to Bangka Island.

Utama’s passion for cycling had led him to open a small bicycle shop and convince Dynatek, a high-end Italian racing-bike manufacturer, to grant him a distributor’s license. He had just finished business dealings with Manfredi, a retired Italian pro cyclist, before they jetted off for a quick vacation to Pangkal Pinang, Utama’s hometown across the Java Sea. The plane went down shortly after takeoff.

Less than three years earlier, Utama had married Friscilla. Their son joined them a year and a half later. The couple celebrated the boy’s first birthday three months before the crash.

“I’m pretty happy, actually, for what he achieved,” Fenlix said of his late brother, who was a year older. “He was a good guy to be the wife of, and he was very proud of their child. He had a huge imagination of what Willfred would become one day.”

After learning about the crash, Fenlix and other family members first rushed to Soekarno–Hatta International Airport west of Jakarta, where dozens of other victims’ family members packed into a “crisis center” to wait for information, he said. They spent the day there, grief-stricken and panicked, but still holding out hope, Fenlix said.

Hours passed without word. Finally, airline officials directed the crowd to go to a different airport, Halim Perdanakusuma International, a 40-minute drive across town. Day turned into night, as several more hours passed. When the airline offered rooms at a nearby hotel to waiting families, Fenlix refused, instead booking his parents into a room at a different hotel. Fenlix left them there and returned to the crisis center, seeking answers.

“There were still many, many people waiting,” he said. “It was crowded and not very well-managed. The only information is that ‘this plane has crashed.’ “

Fenlix watched a video posted on social media showing boats circling debris floating in the sea where the plane had crashed. It was then,  he said, that he finally realized: There were no survivors.

Airline officials eventually directed family members to a police hospital near the airport, where they could provide DNA samples to help identify crash victims. Fenlix brought his father there so investigators could swab the inside of his mouth. Over the next few days, Fenlix gathered and turned over other items: an unwashed bottle his brother had drunk from, identification documents, photographs.

As the days passed, Fenlix and his parents, all devout Catholics, worried what might happen if Utama’s remains were never found. Would the church in their hometown forbid them from holding a funeral? Could a marker even be placed in the church cemetery?

“I told my father that if they don’t allow us to make the grave, then we will still make it,” Fenlix said. “We need to make something so my brother’s son and his wife can pray, so we can have a place to have a memory about him.”

In the end, the family didn’t have to challenge the church. Utama’s remains were recovered, identified and laid to rest.

To mark the somber anniversary of Utama’s death, Fenlix and his family will return to the cemetery.

“We will pray for my brother at his tomb,” Fenlix said.

To get there, Fenlix and his wife, pregnant with their first child, have purchased tickets on a flight from Jakarta to Pangkal Pinang — the very trip his brother never got to finish. They booked the flight with Citilink airline on an Airbus A320 jet, Fenlix said.

Clarification: This story and a cutline for an accompanying photograph described Verian Utama’s friend and fellow Lion Air Flight 610 crash victim Andrea Manfredi as a business partner. Manfredi was a supplier of Italian bicycle accessories to Utama’s bicycle shop in Indonesia, but he did not have an ownership stake in Utama’s business.