🚀 go-pugleaf

During most of the 80's, I was in charge of the Command Modulator
firmware on NASA's Deep Space Network at JPL-Pasadena. I have stories
galore from there, but I've always especially enjoyed this one. A
story of a vanishing bug.
 
Most of the network at each DSN site consists of interconnected
ModComp II 16-bit minicomputers (we're talking TTL chips here,
folks!).  A recurring problem on one particular subsystem was that it
would occasionally lose data blocks (packets, in modern parlance)
during exchanges with its neighbor subsystems.
 
Since this was not a critical problem (lost blocks were tagged and
retransmitted--where they always got through) it was not a high
priority fix in the midst of upgrading the network for things like
Voyager's Uranus and Nepune encounters and Galileo.  It was,
nonetheless, annoying to the programmer in charge of the subsystem
(we'll call him Ray); try as he might, he could not find out what was
wrong or even reliably reproduce the problem.  The only thing he
noticed was that as time went on the problem would gradually happen
more often, and then would go away again and not happen at all for an
equally long time.
 
One day, long after everybody had gotten used to this behavior (you
had to monitor the packets to even see it at all).  Ray discovered the
bug.  We were all properly amazed as it was only by chance that it
had not bitten everyone a lot worse than it had:
 
Since there were a lot of ModComp II's running on the network, a large
amount of "common software" had been written back when they were
installed in the 70's. Most of this was essentially a library of
custom assembly routines you could use in your own code to keep things
standard.
 
Each routine had some standard documentation written for it, one
item always being the amount of time the routine would take to run
under worst-case and best-case conditions.  A certain long-gone
programmer had written a routine that would perform a "long-divide".
That is, divide a 64 or 32-bit integer by another.  The documentation
page for this routine said that the routine would take between 1.5 and
3 microseconds to execute, depending on the arguments.
 
One convention on the network was that date/time stamps were stored as
a single 32-bit integer.  Said integer represented the count of the
number of deciseconds since midnight, January 1 of the current year.
For his subsystem, Ray needed to divide this value by another, so he
used the aforementioned routine.
 
It turned out that the actual worst-case time for this routine was 5
MILLIseconds--three orders of magnitude slower that the best case
time!  Furthermore, the variation was a discontinuous function of the
number of 1 bits in the dividend.
 
You can guess the rest. Whenever Ray called this routine, and the
current date/time had a lot of 1's, the ModComp would briefly slow to
molasses speed. If a block happened to be incoming at the time, many
bytes (sometimes the whole block) would be missed.  By the time the
network noticed and sent the block again, Ray's machine would be long
past the bottleneck and the block would read fine.  At various times
more 1's would pile up in the high-order bits of the date/time integer
and slow the system down more frequently.  Later, the bits would carry
to a small number of 1's again and the problem would vanish.
 
Oh, and why was Ray the only one who had this problem? It seems out of
the whole net, he was the only one who called this particular divide
routine.