Python script to download and install EXOS image multiple times

Below Python script and UPM configuration could be used to try upgrade on the switch multiple times.
This script used to make switch download and install EXOS image multiple times.

[upgrade.py]

import time

import re

import sys

def upgrade():

for loop in range(1, 51):

upgradeCMD = 'download image x.x.x.x summitX-16.2.3.5-patch1-3.xos vr
"VR-Mgmt" primary'

result = open('/usr/local/tmp/upgraderesult.txt', 'a')

cmdOutput = exsh.clicmd(upgradeCMD,capture=True)

result.write(cmdOutput)

time.sleep(10)

result.close()

if __name__ == '__main__':

   try:

       upgrade()

   except SystemExit:

       pass

[configuration]

create upm profile testprofile

load script upgrade.py

.

configure upm profile testprofile maximum execution-time 590

create upm timer testtimer

configure upm timer testtimer profile testprofile

configure upm timer testtimer after 10 every 900

TCL script to download and install EXOS multiple times

This is an UPM script using TCL and configuration on EXOS to trigger this UPM script. The purpose of this script is upgrading EXOS repeatedly to see whether there is any issue during EXOS upgrade.

[ upgrade.xsf ]

set var temp $TCL(after [expr 60 * 1000])

set var cli.out " "

show switch | in Image

set var output $TCL(split ${cli.out} "\n")

set var line $TCL(lindex $output 1)

set var split0 $TCL(split $line ":")

set var partition $TCL(lindex $split0 1)

set var TF $TCL(lsearch $partition *primary* )

if ($TF) then

 download image 10.67.55.52 summitX-15.6.4.2-patch1-6.xos vr "VR-Mgmt" primary

elseif

 download image 10.67.55.52 summitX-15.6.4.2-patch1-6.xos vr "VR-Mgmt" secondary

endif

reboot

[ config ]

create log filter testfilter

configure log filter testfilter add events HAL.Card.Info match string "Switch is operational"

create log target upm test

configure log target upm test filter testfilter severity Info

enable log target upm

create upm profile test

load script upgrade.xsf

.

configure upm profile test maximum execution-time 600

configure vlan mgmt ipaddress y.y.y.y

configure syslog add x.x.x.x:514 vr VR-Mgmt local1

enable log target syslog x.x.x.x:514 vr VR-Mgmt local1

Troubleshooting OSPF neighbor flapping by TCL script on EXOS

If OSPF problem is existing in customer’s network, we could narrow down the cause of this issue by the log filter and access-list, etc.

( whether ospf hello message dropped on sender side or receiver side, if it is receiver side then, whether is dropped on slow patch or it is not found on HW etc )

Please see below example and result in the lab.

1. log filter

[config ]

configure log filter DefaultFilter add events ospf.hello.PktRx match ipaddress "1.1.1.2"

[result ]

* cntcom01.75 # sh log

11/21/2017 13:44:46.89 <Info:ospf.hello.PktRx> MSM-A: test received a Hello pkt from 1.1.1.2

11/21/2017 13:44:36.88 <Info:ospf.hello.PktRx> MSM-A: test received a Hello pkt from 1.1.1.2

2. access-list

Policy: ospf1

entry e1 {

if match all {

   source-address 1.1.1.2/32 ;

   destination-address 224.0.0.5/32 ;

}

then {

   count ospfc_1.1.1.2 ;

}

}

entry e2 {

if match all {

   source-address 1.1.1.3/32 ;

   destination-address 224.0.0.5/32 ;

}

then {

   count ospfc_1.1.1.3 ;

}

}

configure access-list ospf1 vlan test

3. UPM

[ ospfcount.xsf ]

set var cli.out " "

show access-list counter

clear access-list counter

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var total ($len - 6 )

set var cnt 0

set var ospflineindex $TCL(lsearch $output *ospfc* )

while ($cnt < $total) do

           set var line $TCL(lindex $output $ospflineindex)

           set var ospfcount $TCL(lindex $line 1)

           create log entry OSPF_COUNT:$ospfcount

           set var cnt ($cnt + 1)

           set var ospflineindex ($ospflineindex +1)

endwhile

[config on switch]

create upm profile ospfcount

load script ospfcount.xsf

.

create upm timer ospfcount_timer

configure upm timer ospfcount_timer profile ospfcount

configure upm timer ospfcount_timer after 10 every 60

[result]

11/21/2017 14:46:34.99 <Info:System.userComment> OSPF_COUNT:6

11/21/2017 14:46:34.97 <Info:System.userComment> OSPF_COUNT:6

11/21/2017 14:46:34.86 <Noti:UPM.Msg.upmMsgExshLaunch> Launched profile ospfcount for the event UPM Timer ospfcount_timer

11/21/2017 14:45:34.35 <Info:System.userComment> OSPF_COUNT:6

11/21/2017 14:45:34.33 <Info:System.userComment> OSPF_COUNT:6

11/21/2017 14:45:34.20 <Noti:UPM.Msg.upmMsgExshLaunch> Launched profile ospfcount for the event UPM Timer ospfcount_timer

4. on neighbor switch ( for example, on 202.251.2.48 )

configure log filter DefaultFilter add events ospf.hello.TxPkt

enable log debug-mode

[result]

* cntcos01.77 # sh log

11/21/2017 14:49:57.24 <Summ:ospf.hello.TxPkt> MSM-A: test is transmitting a Hello pkt to 224.0.0.5 (length 56, interface 0x0050B638, pkt num 8293).

11/21/2017 14:49:47.24 <Summ:ospf.hello.TxPkt> MSM-A: test is transmitting a Hello pkt to 224.0.0.5 (length 56, interface 0x0050B638, pkt num 8292).

Python script on EXOS - Ping test to Backbone device and disable the port if ping failed

EXOS support python script from 15.6. Below script is one of example can be used on EXOS.
In below example, switch will test ping every 10 seconds and disable uplink ports and generate the log if ping to backbone device failed.

# vi pingCheck.py

import exos.api

import time

import re

import sys

lagPort = sys.argv[1]

ip_addr_list = [(sys.argv[2], sys.argv[3]), (sys.argv[4], sys.argv[5])]

timeSleep = 10

def pingCheck(ip_addr, vr_name):

   pingCMD = 'ping count 3 vr {} {}'.format(vr_name, ip_addr)

   cmdOutput = exos.api.exec_cli([pingCMD]).splitlines()[-2]

   pingResult = re.search(r'received,\s+(\d+)%\s+loss', cmdOutput).group(1)

   return True if pingResult == '0' else False

def lagCheck(lagPort):

   lagCMD = 'show lacp lag {} | include {}'.format(lagPort, lagPort)

   cmdOutput = exos.api.exec_cli([lagCMD]).splitlines()[0]

   lagResult = re.search(r'\s+(\d+)\s+(\w{2}:?){6}', cmdOutput).group(1)

   return True if int(lagResult) > 0 else False

if __name__ == '__main__':

   result = True

   while True:

       if result:

           for ip_addr, vr_name in ip_addr_list:

               result = pingCheck(ip_addr, vr_name)

               if not result:

                   runCMD1 = 'create log message "WARNING!! Ping failure to {}, Disabling the LAG port ({}) to upstream Cisco router"'.format(ip_addr, lagPort)

                   runCMD2 = 'disable ports {}'.format('85-96')

                   runCMD3 = 'create log message "Process PingCheck will stop until the LAG port {} is manually re-enabled"'.format(lagPort)

                   exos.api.exec_cli([runCMD1, runCMD2, runCMD3])

                   break

       else:

           result = lagCheck(lagPort)

           if result:

               runCMD = 'create log message "The LAG port {} has been manually re-enabled and brought up. Process PingCheck has started again"'.format(lagPort)

               exos.api.exec_cli([runCMD])

       time.sleep(timeSleep)

From EXOS version 15.7, user can create process running based on user created python script. With below configuration on the switch, process will be created to run the script named pingcheck, and will use 4 values argument.

# create process "pingCheck" python-module "pingCheck" start auto "85" "115.164.3.131" "VR-accessPODSHT" "115.164.3.163" "VR-accessPODSHT"

* 85 is the lag port to disable when failure is detected. "115.164.3.131" "VR-accessPODSHT" is the ip address and vr name to monitor by ping.

TCL Script for Ping test and check result

Below script can be used to ping multiple hosts and clear fdb, and disable the script if ping loss happened. With this script, switch will ping from 172.27.77.201 ~ 172.27.77.210 and if more than 5 hosts show 100% ping loss, then it will stop the UPM profile named Test_SK.

set var 77ServerNum 201

set var i 0

while ($77ServerNum < 211) do

set  var cli.out " "

ping count 10 172.27.77.$(77ServerNum)

set var split_list $TCL(split ${CLI.OUT} "\n")

set var percent $TCL(lsearch $split_list *%*)

set var lossline $TCL(lindex $split_list $percent)

set var loss $TCL(lindex $lossline 6)

if (!$MATCH($loss,100%)) then

create log message "172.27.77.$77ServerNum => Not Connected"

set var i ($i + 1)

endif

set var 77ServerNum ($77ServerNum + 1)

endwhile

if ($i <= 5) then

create log message "ping failed on $i hosts"

clear fdb vlan ToR

endif

if ($i > 5) then

disable upm profile Test_SK

endif


To execute the script periodically, upm profile and timer need to be configured on the switch. In below example, upm profile named Test_SK is configured to execute the script named pingtest and upm timer named Test_SKTimer configured to execute the profile named Test_SK every 240 seconds.

create upm profile Test_SK load script pingtest.xsf . configure upm profile Test_SK maximum execution-time 200 create upm timer Test_SKTimer configure upm timer Test_SKTimer profile Test_SK configure upm timer Test_SKTimer after 1 every 240

script to check packet count to CPU

Below script gather the infrormation of “show ipstats” regualraly ( every 10 sec ) and and generate the log if the number of packet to cpu is over 100.

disable clipaging

clear counters ipstats

set var temp $TCL(after [expr 10*1000])

set var CLI.OUT " "

show ipstats

set var flr_list $TCL(split ${CLI.OUT} "\n")

set var len $TCL(llength $flr_list)

set var ex_cnt $TCL(lindex $flr_list 1)

set var count2cpu $TCL(string range $ex_cnt 17 22)

if ($count2cpu > 100) then

create log entry "the count is over threshold"

disable port all

endif

UPM script on EXOS

Below script is an example of UPM script, which can be used to check the access-list configured on each IO module and reconfigure ( clear and configure again ) the ACL. UPM script can be triggered by log message of EXOS switch by using log filter. In below script, whenever slot rebooted, it generate the relevant log and this log trigger the UPM script to be activated.
Below script used as a workaround of OS bug. If problem found on OS, it takes some time to clarify the Root cause of Bug and fix the problem on OS and release fixed version. So, even it's not the final resolution but using script as a workaround and minimizing the service impact  is always good idea.

[ UPM configuration ]

#

# Module upm configuration.

#

create upm profile acl1

set var cli.out " "

show access-list | in 1:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl10

set var cli.out " "

show access-list | in 10:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl2

set var cli.out " "

show access-list | in 2:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl3

set var cli.out " "

show access-list | in 3:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl4

set var cli.out " "

show access-list | in 4:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl7

set var cli.out " "

show access-list | in 7:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl8

set var cli.out " "

show access-list | in 8:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

create upm profile acl9

set var cli.out " "

show access-list | in 9:

set var output $TCL(split ${cli.out} "\n")

set var len $TCL(llength $output)

set var cnt1 ($len - 1)

if ($cnt1) then

set var i 0

while ($i < $cnt1) do

set var line $TCL(lindex $output $i)

set var port $TCL(lindex $line 1)

set var rule $TCL(lindex $line 2)

set var direction $TCL(lindex $line 3)

unconfigure access-list ports $port

configure access-list $rule ports $port $direction

set var i ($i+1)

endwhile

endif

.

[ EMS configuration - log configuration ]

create log filter aclfilter1

create log filter aclfilter2

create log filter aclfilter3

create log filter aclfilter4

create log filter aclfilter7

create log filter aclfilter8

create log filter aclfilter9

create log filter aclfilter10

configure log filter aclfilter1 add events HAL.Card.Info match string "Slot-1 is operational"

configure log filter aclfilter2 add events HAL.Card.Info match string "Slot-2 is operational"

configure log filter aclfilter3 add events HAL.Card.Info match string "Slot-3 is operational"

configure log filter aclfilter4 add events HAL.Card.Info match string "Slot-4 is operational"

configure log filter aclfilter7 add events HAL.Card.Info match string "Slot-7 is operational"

configure log filter aclfilter8 add events HAL.Card.Info match string "Slot-8 is operational"

configure log filter aclfilter9 add events HAL.Card.Info match string "Slot-9 is operational"

configure log filter aclfilter10 add events HAL.Card.Info match string "Slot-10 is operational"

create log target upm acl1

enable log target upm acl1

configure log target upm acl1 filter aclfilter1 severity Info

create log target upm acl2

enable log target upm acl2

configure log target upm acl2 filter aclfilter2 severity Info

create log target upm acl3

enable log target upm acl3

configure log target upm acl3 filter aclfilter3 severity Info

create log target upm acl4

enable log target upm acl4

configure log target upm acl4 filter aclfilter4 severity Info

create log target upm acl7

enable log target upm acl7

configure log target upm acl7 filter aclfilter10 severity Info

create log target upm acl8

enable log target upm acl8

configure log target upm acl8 filter aclfilter8 severity Info

create log target upm acl9

enable log target upm acl9

configure log target upm acl9 filter aclfilter9 severity Info

create log target upm acl10

enable log target upm acl10

configure log target upm acl10 filter aclfilter10 severity Info