[evla-sw-discuss] Priority 1 IOS problem

[James Robnett]

   We have a very severe problem with the new IOS we just
discovered.

   Most of the antenna modules (MIBs) have their address hard
coded but 4 use DHCP to figure out their network.  The first
antenna to be power cycled since the IOS upgrade occurred yesterday
and those 4 MIBs never came back up.

   Non-production test MIBS also use DHCP.

   I can see test MIB's at the site successfully booting right
up until the day we upgraded the IOS.  Since that day no test
MIB's that use DHCP have successfully booted.   Sadly nobody
ever reported a problem.

   The new IOS is somehow modifying the reply either by blocking
it or actually changing the packet.  Routers definitely wrap
the DHCP packets for forwarding.  It's how the DHCP server knows
what network the request originated on.

   The ISC DHCP server is getting the initial DHCP request and
is sending a reply.  All non-MIB's that do DHCP work just fine
so there's some nuanced change in how the new IOS is treating the
replies that is valid for new DHCP specifications but not for
older ones, in fact this may even technically be bootp.

   We have antenna moves scheduled starting next monday.  If
we can not figure this problem out by today we'll have to revert
to the old IOS or risk losing every moved antenna.

   Derek we need to create a new priority 1 TAC with Cisco, make
certain they have my office phone (7226) and cell (575-418-7368).
Office is preferable but after lunch today I'll carry my cell with
me.

   I'll start looking for known DHCP problems with this IOS.

james
ps: Reverting the IOS will remove the alert packet fix but should
not effect reference pointing.  We will need to test that pretty
carefully if we do in fact revert.