Don't rush the testing process
Rigorous Review
"maddog" recounts a tale illustrating the importance of thorough product testing.
The newest Free and Open Source exploit is called Shellshock, and I hope it will finally illustrate that GNU/Linux is not free of bugs, any more (or less) than closed source code is free of bugs.
However, I've seen several articles in which columnists continue to write as if every piece of open source code has many eyes looking at it (which is not always true) or as if every piece of closed source code goes though rigorous design review and testing by legions of engineers (equally untrue). The answer, as always, lies somewhere in the middle.
Yes, some companies have good design and code reviews that proceed to a field test or beta release that allows end users to test and submit bug reports before the product is unleashed on the general public. Even in the largest companies, however, some projects receive little engineering consideration. The code is written once by engineering, then patched and expanded over time according to bug fix requests and feature requests from end users. Over time, fewer engineers work on each part, with fewer "eyes" on the code.
When I worked at Digital Equipment Corporation (DEC), we were in the process of going to field test with a brand new operating system and a brand new piece of hardware (including a new tape drive called the TK50), when I found out that the field-test software had never actually been tested on the new hardware. I raised the flag in the field test meeting, but people did not want to hold up the start of testing because it would look bad to upper management. Finally, the Senior Vice President of that workstation line called me and asked what was going on. I told him that the rush to bring the product to market was making people take shortcuts that I did not approve of, including the fact that the field test documentation was incomplete and not reviewed. He listened carefully and said he would attend the next meeting and that – no matter what was said in the meeting – we would not be going to field test that week.
The VP came to the meeting and listened while everyone gave their reports, then he asked each person, "Are we ready to go to field test?" Each speaker said "yes." Finally, he came to me. I gave my report and said that I was not happy that each report had some caveats, but I could not second-guess the rest of engineering and I would go along with what other people said.
The VP then stood up and said, "I have to get back to headquarters, but I want to tell you that we are not going to field test tomorrow. You have all been doing a good job, but I feel it is much too rushed, and I want you to take another week."
Some of the engineering managers started to object, but the VP said, "There is no use arguing this, it is my decision. Ken Olsen (at that time the President of DEC) and I both feel you have made your commitment to schedule, but we are going to give it a week to mature." Immediately after that, the documentation manager left the room to tell his people to continue developing the documentation.
That meeting was on a Friday. On Monday, I went to the manufacturing site where the 15 field test units were waiting to be shipped. I tried to install the field-test software on the hardware, planning to wipe the hard disk clean after testing. Thirteen of the 15 systems would not even install, and the two that did install had other hardware problems.
I called over the QA manager of the field test and asked how these systems had passed final diagnostic testing. "They did not pass," he said, "but the main engineering group said the diagnostics were wrong and just to ignore the results of the diagnostics."
Fuming, I went back to central engineering and asked what was going on. "Oh no!" declared the diagnostic engineer, "ONE of the diagnostics was bad, and I told them to ignore the result of that ONE diagnostic. The other 368 diagnostics were fine, and he should have been flagging any of those that failed." The engineering team then did an evaluation of why every single one of the field test systems had failed manufacturing and fixed them before they left DEC.
If we had gone to field test according to schedule, we would have lost time and money fixing these issues out in the field, as well as credibility in the eyes of the customer.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.