Who is affected?

A lot of users will be affected . Anything goes through firewall and datacenter service will be down during this time. This work will be disruptive.

Problem:

SRX and QFX upgrade is scheduled on 10/30/2016

Systems Involved:

Network

Details:

Starting 3am on Sunday, October 30th,  Security team will upgrade SRX update first and it will take about 30-60 minutes. Network team will upgrade QFX after that and will take probably 2 hour.  Everything will recover before 6am if everything goes smooth.

Updates:

10/30/2016 11:34:05 AM:
11:35 AM The issue was not link aggregation but that auto-negotiation was disabled in the upgrade code. This was a change from the previous version and all BaseT ports in the Data Center were impacted. The workaround was to manually configure auto-negotiation as explained in PR1170909. At this time we believe all issues are resolved. We also believe that while many systems were impacted for most service was degraded not lost.

10/30/2016 9:11:36 AM:
9:10 AM Upgrade is complete. Packect loss has ceased. An issue specific to link aggregation has caused some systems to lose connectivity. A workaround is being implemented and a fix for link aggregation is being determined with support.

10/30/2016 7:53:49 AM:
7:50 AM Twelve of sixteen switches complete. Packet loss inside of the Data Center is occuring during the upgrade. The upgrade will be in progress after 8 AM. Once the upgrade completes we'll evaluate the packet loss to determine whether it is ongoing or isolated to the upgrade.

10/30/2016 6:53:38 AM:
6:50 AM Seven of sixteen Data Center switches have upgraded successfully. Work continues on the remaining nine.

10/30/2016 4:24:41 AM:
4:00 AM Firewall upgrade is complete. Data Center has begun

Problem Resolution: 

At this time we believe all issues are resolved and the Data Center is fully functional.