SupportBig Money Engineering Integrity
Every now and then, the ZipCPU blog includes a short discussion on engineering ethics. Our last post challenged you with the question, what would cause you to lie? Below is a discussion from some of my own experience, illustrating the extent to which an integrity problem can destroy the foundation of solid engineering.
Before I started Gisselquist Technology, I had my own moral strength and integrity challenged.
Many years ago, I was asked to join the government team managing a multibillion dollar contract. To protect the innocent, we’ll leave the name of the project and the name of the office supporting it nameless. Suffice to say that I was working for a government procurement office.
The director of my office asked me to examine the product that he was responsible for procuring on behalf of the US government. I was not asked because it was my current job (I had been working on another team), but rather because of some heated disagreements over whether or not the product actually worked. Since my academic background was in a nearly related field, I was deemed to be uniquely suited to evaluate the product that was being produced.
What my research revealed
As I started researching this program, its requirements and history, I quickly discovered that the history of the program was quite colored.
 |
Interviews with the contractor revealed a history of employees getting re-assigned or fired for telling the government procurement lead that the product didn’t or wouldn’t work. Those that remained, the ones I was able to talk to, were very aware that remaining on the contract, indeed their very job and livelihood, would only be possible if they said nothing about any problems the system had.
Those who worked for the contractor were both scared and morally scarred. They were living under a lie and they knew it and didn’t like it. However, they felt trapped. If they spoke up, they knew that they lose their jobs and how would they then feed their family?
The corruption wasn’t limited to the one office. Rather, it spread throughout the contractor’s facility, throughout the government procurement office, and even into the leadership of those who were supposed to use this product.
For example, all of the official program briefs presented glowing numbers showing the amount of data the system could process. The quantity was deemed to make the program a success. The program manager was very quick to display all kinds of figures showing the massive amounts of data this system produced. However, he had no means of proving that any one of those outputs were in fact correct.
 |
The actual user’s of the program, on the other hand, would present example upon example demonstrating the failures of the program. However, without maintaining the data set that created those failures, the failures could never be reproduced. Since the failures couldn’t be reproduced, they couldn’t be used as test cases and formed ambiguous requests for improvement within the program office. Further, since the system was never tested against known inputs in conditions similar to those that had caused known failures, it was impossible to know if the failures were common behaviors, or just the one in a million rare example.
The result was that any time the two offices got together, the meetings could easily devolve into shouting matches. It works! It’s broken! With no truth under-girding either argument, the two sides might just as well have been arguing: Taste’s great! Less filling! There would have been no difference in the result.
This is the sad result of what happens when sound engineering practice is abandoned.
Rather than treating this as the integrity problem it was, the problem was officially characterized as “political”. By management fiat, I was instructed and told that the problem had nothing to do with sound engineering practice.
However, the problem was not “political” as my boss had instructed me. The problem was the result of the loss of the Godly integrity that forms the foundation of sound engineering practice.
 |
In the digital design world, we might say that the project didn’t include any representative test-benches. How was it tested, you ask, without test-benches? By comparing the system’s results before any software changes to the results after the improvements. But … which of those results were actually correct? No one knew.
No one was placing known inputs into the device to measure whether or not it was producing the correctly expected outputs. This is the idea behind a “ground truth” test. Known inputs are measured against what should be known outputs. Differences are recorded, quantified, and reported on.
Ground truth testing
 |
A “ground truth” testing program that compared the output of this product against an expected or “correct” output was called for. Further, a library of “ground truth” test cases having known inputs needed to be assembled that would highlight any problems, and also allow the development team to fix them. The program basically needed test benches with physically realistic inputs.
With this goal in mind, I quickly assembled a small team and secured funding for a project that would help to determine how the system actually performed against known inputs. We called it a “ground truth” testing program. To my surprise, the idea of “ground truth” testing program was not universally acclaimed as good engineering. Indeed, it was my own office that considered the idea “controversial.” I was told that such testing would be a waste of precious dollars, and that it wouldn’t prove anything to those that “didn’t like” the system.
While this “ground truth” testing program was underway, a senior leadership “taskforce” was appointed to fix all the system’s problems. Management gave them ninety days to be successful. Reality, though, is that years of engineering integrity problems are not easily fixed in ninety days.
 |
Think about that one: what would you do if you were the one appointed to a ninety day task force, and tasked with fixing problems that were the result of years of integrity neglect?
While I was not personally involved in this “taskforce”, I read with great interest their results and conclusions.
Much as one might expect, the “taskforce” declared the system fixed. Had they compared the output of the system against known inputs? No.
Did the “task force” collect examples of inputs causing “known failures”, apply fixes, and then run those same examples again to see if the result got better? No. They applied fixes and tested the system with new data which may or may not have had the original problem within it. There was no way to know if the fixes even worked. Indeed, all those who weren’t part of the senior leadership team were instructed to accept that the problems had been fixed–independent of any real facts.
This is not how you fix such a system.
What the “task force” should have done would have been to first find data sets that would have reproduced the problems, and then replayed those data sets through the system later to see if any of the “fixes” worked. Further, their data sets needed to come with known answers–such as the “ground truth” testing approach was working to create.
When I saw the results of this senior leadership “taskforce”, I spoke up. I sent an internal memo to the rest of the procurement team outlining how the process was broken. I explained that the senior leadership “taskforce” hadn’t demonstrated that they had fixed anything, since they had abandoned sound engineering practice. As I’m sure you can imagine, my office manager wasn’t pleased with my views. When he read this memo, he then called me into his office and threatened me. “You pull another stunt like this, and you will be fired.”
During this time, the ground truth testing effort continued.
To illustrate how bad the environment was at the contractor’s facility, they were making bets over when the ground truth project was going to be killed by my management (in the procurement office).
The procurement management team never liked the “ground truth” testing approach, probably because they didn’t feel like they could control it’s results. Measurements are pesky things, and comparisons between system outputs and known “correct” answers are hard to spin. Indeed, the “ground truth” testing project was nearly terminated once when the procurement manager, whom I worked for, realized what I was up to. (It was also nearly terminated when he read the memo I referenced above.)
As you might expect, testing a product in a way that it has never been tested before will likely yield unexpected results. This is normal. This was the result of the “ground truth” tests.
The product itself was not irreversibly broken. Sound engineering process, built upon true engineering and integrity, would’ve fixed it. The real problem was that no one was using sound engineering principles because of the lies and the fear involved. True God given engineering integrity had been lost.
Sure enough, when the procurement manager read the “ground truth” report, he didn’t like its results. He then directed me to withhold these results from the people who were going to use this product.
The aftermath
This was about where the office’s Inspector General (IG) got involved.
This was also the point where I learned the legal definition of the word “fraud”.
Fraud is generally defined in the law as an intentional misrepresentation of material existing fact made by one person to another with knowledge of its falsity and for the purpose of inducing the other person to act, and upon which the other person relies with resulting injury or damage. Fraud may also be made by an omission or purposeful failure to state material facts, which nondisclosure makes other statements misleading. uslegal.com
Let’s break this down into simpler terms. I had created for my office a “material fact”: the ground truth testing report. By withholding this fact from the operations office (fraud by omission) that was supposed to accept our product, they had suffered “damage”: they were forced to accept as operational a system that never passed a valid test. This met the legal definition of fraud.
Following the IG’s report, the program had a change of leadership. Some individuals lost their jobs. Several individuals were quietly forced to retire. I was also given my one and only chance to brief the US Congress. I would have rather had my opportunity under more pleasurable circumstances. However, I knew my duty.
I also chose to leave that community and seek other work, so as to avoid any ethical or whistle-blower issues.
Since then, I have kept up with members of contractor’s office. I have thus kept track of what has happened after my departure. A new management team has been assigned to replace the old. The new management team, though, kept the same advisors from the last team. The sad result is that the old advisors convinced them to follow the same crooked path that had led them into the mess in the first place. As a result, this product is still built upon lies, having no sound engineering integrity within it.
Breaking the Liar’s Dilemma
Throughout this whole process, I would constantly explain to my team that your actions should all depend upon your judge. If the boss is your judge, do as he tells you to do. If the IG or Congress will be your judge, act so as to impress them. On the otherhand, if anticipate that God will be your judge, then don’t worry about the boss, the IG, or Congress: focus your life on pleasing God.
What about the liar’s dilemma?
Jesus provides some clear guidance:
Therefore take no thought, saying, What shall we eat? or, What shall we drink? or, Wherewithal shall we be clothed? (For after all these things do the Gentiles seek:) for your heavenly Father knoweth that ye have need of all these things. But seek ye first the kingdom of God, and his righteousness; and all these things shall be added unto you. Take therefore no thought for the morrow: for the morrow shall take thought for the things of itself. Sufficient unto the day is the evil thereof. (Matt 6:31-34)
Indeed, this guidance breaks the trap of the liar’s dilemma. In this verse, Jesus promises that God will provide and all we need to do is to “seek first the kingdom of God”. The big problem with the liar’s dilemma is that those caught within it believe that their life will be worse, if not even destroyed, if they do not lie. Yet for those willing to trust Jesus at His word, they will learn that this is never the case.
I read these verses many times during my work on this project.
Yes, I have left that job and that twisted work environment. No, I am not making the “big bucks” today, but I did manage to leave with my integrity. Thanks be to God.
Better is the poor that walketh in his integrity, than he that is perverse in his lips, and is a fool.