🚀 go-pugleaf

jcmo...@mitre.org (Joe Morris) writes:
> 
> Of course, *nothing* brings out the bugs like putting a system into
> production...no matter how thoroughly you test for broken responses
> to pathological conditions, some user will find a new way to expose
> a failure mode.
> 
> Joe Morris

misc. refs: from disk controller testing
http://www.garlic.com/~lynn/96.html#18
http://www.garlic.com/~lynn/96.html#19
http://www.garlic.com/~lynn/subtopic.html#disk

basically the "acceptance" test for 3880 was that it was within +/-10%
of the 3830 performance. the problem was that while the 3880
supporting data streaming 3mbyte transfers ... it did it with custom
hardware for the data path and a jib' microcontroller for the control
path (the 3830 had been all a fast horizontal microcoded engine). The
jib' was taking quite a bit longer to do clean-up after an operation
finished ... so to meet the performance criteria ... they started
presenting operation finished interrupt "early" (before the jib' had
finished all its clean-up). This worked OK for the acceptance test
which was with a one-pack/drive, single task VS1 system.

The problem was the weekend the product test lab replaced a production
16-drive 3830 configuration with one of the 3880 controllers. I got
the call bright and early monday morning to come fix a operating
system problem (performance had all gone to $%#!). Finally was able to
trace it back to the "new" 3880. Fortunately this was six months
before first customer ship ... and allowed the engineers some time to
make other adjustments to address the issues.

The problem was that with multiple drives (and high level of
multitasking) on the same controller, there were frequently requests
queued because of channel, controller, and/or device busy. When the
controller signaled operation complete interrupt ... the operating
system would immediately hit/redrive it with a queued request.
Because the jib' had signaled "complete" early, it was actually still
busy ... and so the redrive operation got a "control unit busy"
response. The system then went off to do something else. Eventually,
the jib' finished and because it had signaled a control unit busy
condition, it then had to signal a control unit end interrupt. With
this, the system the attempt to restart the redrive operation (which
would typically succeed this time).

The extra operation and interrupts .... in addition to the increase in
operation latency made things look like a 80-90% system degradation.

-- 
Anne & Lynn Wheeler   | ly...@garlic.com -  http://www.garlic.com/~lynn/