Re: Top Ten Things You'd Like To Tell Engineers

[written_by -at- juno -dot- com]

> > I would ask (demand?) that engineers to spend a few weeks on the
> > production floor, assembling the products they expect others to 
> assemble.
> 
> Do you do the same?

Hi TechComm Dood... long reply

We tried to involve engineering, but most of those folks did not want  to
come out and play with us. We were the unclean masses. They were often
summoned to the production floor when problems occurred, but for the most
part, there was very little interaction between engineering and
production. 

Many of the engineering folks were also quite absent when a new product
was introduced, or they would make a quick appearance and disappear. Not
all of them, just most of them. They were specifically required to spend
the day monitoring the line when we ran a Production Verification Test
(PVT). 

Consider our In-Circuit Testing equipment. An engineer was required to be
on the production floor if we had 4 panels fail in a row for the same
issue. Like "open at r45." The component would be glanced at and if it
looked ok, it had to be a fake fail. Problem was, the component was
perhaps cracked or there was PCB contamination, so it failed. Perhaps the
ICT was looking for a 100k resistor and found a 10,000k instead.  Then
the products fail at final test and had to be reworked.

Had the "fake fail" been taken seriously, the engineer would have perhaps
discovered a test program error and fixed the problem. We did not issue
meters to the ICT operators, so it was often a matter of guessing.
Sometimes, the engineer thought that the problem was a bent pin and to
stop worrying about it. Just pass the boards on to the router.

The requirement that when 4 panels fail, an engineer was to be summoned
was set in stone and covered in the very procedures engineering wrote for
us to follow. Eventually, the procedures were changed to eliminate the
need for an engineer to be there when there was a problem. I think we
delayed their tee time, so it had to end.

When you have a full rack of panels fail ICT, there is a problem. The
operators of the ICT equipment actually coined the term "fake fail." This
excuse was used to "pass" a panel that failed ICT testing. Sometimes a
component was missing or oriented incorrectly, so it was perhaps a
management/inspection issue, not an engineering issue. Engineering's
absence really can't be excused, however.

Hand component replacement is expensive. If the component has to be
removed and replaced by hand, there is potential for disaster and damage.
Anyone who has ever had to replace hundreds of fine pitch components will
agree. It takes a special skill to do this quickly and accurately.

By the time the product is reworked time after time to fix a problem,
they look like garbage. Quite often, it is a matter of a bent test pin,
but it is always the job of the engineer to solve the issue and make the
call.  If you have a few old modems or combo cards we built, open them up
and look for yourself. Some products have been reworked more than a dozen
times, and are swimming in rosin flux. Some have had chip sets replaced
3, 4 or 5 times, the same shift that they were manufactured.

Since the PCBs are often broken out of the panel, we could not retest
them using the ICT. There were other problems that would have been
discovered if everyone was paying attention. For example, replacing 20 or
so wrong resistors on thousands of boards, because engineering made a
goof and entered a wrong component value, and we passed the board off as
a fake fail, under their direction. 

Engineering suggested a heat gun be used to remove the fine pitch
components, but the heat units they specified damaged boards, and the
temperatures created often exceeded the chip manufacturer's
specifications. 

They thought it was a good idea, so it came to pass. Engineering thought
many great ideas were just that. Sadly, some of their best ideas cost us
millions of dollars over the years. Some ideas caused us to actually ship
products sans the internal PC Boards.

Sometimes, the heat damaged the 68-pin connector, so even though the
product passed testing and was assembled, the damage was discovered and
the product went to rework, where 60% eventually failed. Sometimes, it
was board delamination problems or parts on the B side falling off or
moving.

Engineers often overlooked the fact that they were dealing with a
problematic production crew who made it up as they went along. Most
engineers never bothered to visit us, even though they were 100 feet
away. Those that were often too lazy to do their job, were quite put out
because they thought that we were too stupid to do ours. 

Eventually, the term 'fake fail' was added to the procedures by
engineering. A huge and costly mistake, as there are no fake fails. It is
either a PCB issue or a fixture / ICT issue. A fail is a fail, and there
is a reason for it. 

According to engineering, we were to call them down only when they had
something like 10 racks of "fake fails." That could represent 200
external modems or  NIC cards; or in the case of some products, 24
individual boards per panel. Engineering would often arrive and see 300
panels of WIP to deal with. 

More than once, half of a shift's production output would be waiting for
them the next day, all were referred to as fake fails. All had to be
looked at individually and repairing them usually caused product loss. If
it was discovered that a wrong part was used, 100 percent of the previous
shift's output had to be pulled from packaging/shipping and fixed.

One duty that for many years was limited to an engineer, was the removal,
cleaning and replacement of the ICT test fixtures. Engineering was
adamant that only they do this, and for good reason. If you damage or
bump a test pin, nothing will pass. Or worse, some part of the test will
be missed altogether. Engineering suddenly decided to allow us to change
fixtures. 

Eventually, anyone could fiddle with fixtures, and it cost us dearly.
When engineering rewrote the procedure, nothing about proper fixture
confirmation was mentioned. Some products passed ICT -not because we made
them properly -but because an incorrect test fixture was used to test the
PCB. 

To change a fixture, one had to shut down the line, remove the old
fixture, open the test script file in a text editor, compare the fixture
serial number to the number in this file, inspect the fixture pin by pin
(perhaps 300 pins) for damage and proper height above the fixture bed,
clean the contacts, install the replacement fixture and run some tests.
When engineering did it, it was deemed important. When an operator does
this one hour job, the supervisor called holding up production. This is
never allowed, so we suffered.

Engineering thought that anyone could change and work with fixtures, even
if it was the employee's first day on the job. Would you want a new temp
with no computer experience opening up test scripts files or loading a
test file? Or worse, dropping a heavy fixture that you only have one of?

When I wrote procedures, I personally tested them on the floor, building
the product step by step, following only what I had written. I was not an
engineer, but at least I knew that my procedures were correct. 

> > I would ask engineers to stop designing complicated mechanisms 
> that are
> > often impossible to assemble. In one version of the Palm VII 
> Wireless
> > PDA, the antenna had far too many parts, and the product was 
> almost
> > impossible to build.
> 
> Was this made known as a design flaw? What was the result of that 
> discussion?

The product was eventually changed, but it was still a poor design. This
was a few months before the contract manufacturer who bought us left Utah
and the division died out. So there was little time to make changes. In
the latter days, the design might have changed again. but I was gone, so
I do not know. 

My Palm VII sits unused, because Palm's wireless service (palm,.net) is
going away. I once removed the antenna to replace it and I could never
get the new one to fit or function properly. This was a user replaceable
part, by the way. It matters little to me as there will soon be no
service, anyway.

The antenna was always a big problem to assemble. Not to mention that it
was quite impossible to properly test the product because of the design.
It would make sporadic contact, so in some cases, the unit would not turn
on when the antenna was lifted. 

> > I would ask engineers to design using standard parts, not custom 
> parts.
> > For example, use standard size screws that can be ordered by the 
> ton, not
> > screws that are custom made and impossible to install without 
> damage.
> 
> Assuming that this is concerning microelectronics, many times to 
> meet
> product functional specifications the design must involve custom
> parts.

The screws were required to hold the plastic case parts (RIO MP3 Players)
together. They did not need to be custom screws. The product could have
been designed to use off the shelf screws. 

Even though special size screws were specified, when the engineers
arrived to solve the big screw problem, they decided to use the "wrong"
size screw in the hope that the larger size would cut into and grab the
plastic. 

Fine by me, until you have to take them all apart and the specified
screws no longer work because the plastic is now stripped or enlarged.
One solution was to use locktite on each screw. Another solution was to
order larger screws just for replacement when a RIO was to be reworked.
The solution was found and implemented, but the drawings, the bill of
materials (BOM) and supporting docs were not changed.

Entire lots of smaller screws were mixed up with the larger replacement
screws, so even more failures occurred. Some tech solved this issue by
changing the screwdriver configuration, and this was approved by
engineering on the fly, without testing. The excess torque would
occasionally damage the threaded plastic parts, so you had rattles and
cases that fell apart.

> > I would ask engineers not to be too specific about things like 
> label
> > placement. Having thousands of PCBs rejected because the label was 
> a few
> > millimeters too high was ridiculous, and there was no need to 
> reject the
> > boards.
> 
> Is that an engineer thing or a TQM thing?

It was up to the engineer. He wanted it done his way. We did it our way
because his way placed the labels too close to a connector that had to be
added and soldered by hand, and that usually damaged the bar code label,
so it could not be read. We also did it our way because strict position
was not called out in our procedures. Just approximations.

So we relocated the label just a bit. This label was required by the test
equipment. If the bar code can't be read, you can't complete the next
step in the manufacturing process, because the operator must first scan
the code to confirm that the product passed the previous step in the
process. 

All serial numbers were recorded in a database. If a product is not
tested, there is no record, so the next step can't be started. In
addition, his way made it very hard to scan. We used hand scanners and it
would often take dozens of attempts to scan, and this frustrated the
assembly wonks. Sometimes the scan entered a product twice.  

All this fuss for a label that can be placed anywhere and will never be
seen by the customer. A total non-issue.

> > I would ask engineers not to be so critical about cosmetic 
> criteria and
> > to understand that what they consider to be a flaw that forces a 
> product
> > to be rejected, is almost always overlooked by the customer. For 
> example,
> > scratches that take a 10X loupe to see on the plastic frames of a 
> PCMCIA
> > product.
> 
> Huh?

We built PCMCIA cards for many different customers. The PCB went into a
plastic frame and the covers were heat clamped to this frame, thus
completing the package. The covers contained a sheet of adhesive that was
melted by the heat clamp. Scratches would naturally show up on the frame
edges when the PC Card is installed into the laptop or the test fixture.
Not to mention, the heat clamps added a scratch here and there. There was
no way to assemble the product scratch free, period.

At so called "pre-final inspection," before we tested the card, there
could be no frame scratches. Even if it took a 10 power loupe to see
them, no scratches. After test, scratches on the frame were acceptable,
however. Scratches were simply unavoidable, and allowed by the existing
cosmetic criteria. Engineering said no, however, even though the
procedures said yes. I wrote them, so I was quite familiar with them.

A big problem was inspection of the covers. According to the procedures,
the inspectors had a generous 15-20 seconds to judge the quality of the
cover. Print quality, dents or fine scratches were of some concern, and
depending on the size and location, some were ok and some were
rejectable. 

Some inspectors were suddenly required by the engineers to spend at least
three minutes per card looking for defects.  Totally unworkable because
of the quantity we had to build. I can properly inspect a package in
about 10 seconds, and with total confidence that if I passed the unit, it
would meet any manufacturer's expectations of quality.

We rejected hundreds of thousands of products because people did not
follow the criteria, or new engineers came on board with their egos in
tow, tossing their ideas our way. Almost without exception, their ideas
were bad ones. I certainly knew more about the problem and what was
allowed than any new engineer we worked with. The rejections were for no
bloody good reason.

Another problem is the longer you spend inspecting, the more issues you
will discover, thus making a long inspection time a contributing factor
to larger reject rates. Give me an hour per card and a tight criteria,
and I'll reject 99% of all products. 

> > I would ask engineers to design assembly fixtures so the assembly
> > technicians do not have to use a micrometer or a guess. Most (in 
> my
> > experience) techs will either read the micrometer incorrectly or 
> perhaps
> > not really care.
> 
> Isn't that a problem with the techs then? Provided the design was 
> from
> sec and not an engineer's whimsy?

Yes and no. Many/most of the techs we hired did not usually speak
English, so just communicating with them was a problem. So we asked
engineering to provide fixtures that took into account the specification
of the PCMCIA package. We had milled go/no go gauges for the routers that
depanelized the boards from the frames, but we could never get fixtures
from engineering to check package specs. In all my years there, it was
rare to see a micrometer being used. Not even by QC. The PCMCIA standards
are quite specific.

The techs eventually made a few outline drawings that engineering said
were ok, but they were hand drawn on the ESD mats by the techs with a
Sharpie, or they used a cut sheet of card stock to measure the size. This
was not an accurate way to measure in millimeters. We had a dozen
"standards" that did not meet the PCMCIA standard. A set of go/no go
gauges would have made a difference.

> > I would also ask engineers not to automatically dismiss 
> suggestions that
> > come from outside their group. Some of these suggestions can be 
> quite
> > brilliant.
> 
> What are the channels for suggestions and are you following the 
> correct ones?

We were not allowed to talk to engineering. We scared them, I think :>)

We had no formal system in place. We did encourage knowledgeable people
to correct the written and controlled procedures. Those who saw a problem
would print a copy of the particular procedure, make the changes, and
complete a form. 

The procedure was then given to Document Control, who passed it on to the
proper engineer and the original author of the procedure for approval and
retraining of the production lines. I would meet with those requesting
changes and we would go over them. 

It was once suggested that we make a slight change to how we feed PCB's
into the SMT machines. The person who suggested it was ignored, which was
too bad. If there is such a thing as a "killer production app" this was
it. Another had a problem with how test scripts were loaded into some
test computers, and wrote a C++ program to solve the problem. It did, but
he was ignored. He did this on his own time.

Suggestions that could have saved money were ignored. People gave up
because it did little good to suggest anything. This did change when we
started paying for suggestions. Then every suggestion box was stuffed.

As the author of more than my fair share of controlled procedures,
revisions seldom came my way. I took over the PCMCIA hardware procedure,
and saw it go from (I believe) "Rev A" to "Rev Z", then it started over
at "Rev AA" to "Rev. ZZ," and then "Rev AA1" on up." The last time I
checked, it was at "Rev. AA6" or so. I seldom saw any of the changes,
even though as the author, I should have. I would have rejected most
changes. 

For example, revising the document to add a single missing period to a
sentence. Or the requirement that all products be photographed, the
defects circled, and the products and scans to be shipped to the QC
department of the company we built the products for. The last thing 3Com,
Dell or Gateway wants is for us to send them ten thousand bad cards and
an equal number of scans.

Bob

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ROBOHELP X5: Featuring Word 2003 support, Content Management, Multi-Author
support, PDF and XML support and much more!
TRY IT TODAY at http://www.macromedia.com/go/techwrl

WEBWORKS FINALDRAFT: New! Document review system for Word and FrameMaker
authors. Automatic browser-based drafts with unlimited reviewers. Full
online discussions -- no Web server needed! http://www.webworks.com/techwr-l 

---
You are currently subscribed to techwr-l as:
archiver -at- techwr-l -dot- com
To unsubscribe send a blank email to leave-techwr-l-obscured -at- lists -dot- raycomm -dot- com
Send administrative questions to ejray -at- raycomm -dot- com -dot-  Visit
http://www.raycomm.com/techwhirl/ for more resources and info.