As an administrator it’s our job to keep the SAN up always and without any errors. But sometimes we see errors of different type and these conditions detoriates SAN performance in a great way. Similarly I saw an error condition on one of our Brocade director class switch. The problem reported was is the following,
“Exchange server connected to slot 2/31 is experiencing huge latency”
I checked the portstatusshow for port 2/31. Everything seems to be ok except the following parameter.
er_bad_os 591691 Invalid ordered set
For this condition the fill word need to be set. But it’s already set to correct fill word which is 3 for 8 Gbps network, but this keeps on increasing. So I checked errdump following is what I observer.
2012/12/24-06:34:16, [C2-1012], 592988, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S3,P-1(51): Link Timeout on internal port ftx=1722774272 tov=2000 (>1000) vc_no=31 crd(s)lost=3 complete_loss:1.
2012/12/24-06:40:12, [C2-1012], 592989, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S3,P-1(51): Link Timeout on internal port ftx=1723089858 tov=2000 (>1000) vc_no=31 crd(s)lost=3 complete_loss:1.
There were numerous of these events occurring at different time interval, As per brocade message reference guide this condition is called and Stuck VC. That means the virtual channel which connects to the ports to condor2 ASIC is stuck. If the virtual channel is stuck then it will not pass any signals hence we see the er_bas_os counters increasing. Also noticed link reset happening on the same port from supportshow output.
To prevent stuckvc condition brocade recommends to enable bottleneckmon – feature of FOS (its turned off by default). Stuckvc is a cause of buffer to buffer credit loss.
Buffer Credit Background and Review
To prevent a target device (either host or storage) from being overwhelmed with frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frames. If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire Fibre Channel sequences needing to be retransmitted across the link. Read more here : http://goo.gl/OzMaF9
So now its evident that we have and stuckvc condition. Following is the command with suitable argument to enable bottleneckmon
bottleneckmon –cfgcredittools -intport -recover onLrOnly
This is a non disruptive command so noting to worry of.
What it does? – “Once enabled the agent will monitor the internal links and if there is a 2-second window without any traffic on a backlink with a stuck VC, it will reset it to solve the stuck VC. This approach minimizes the impact of the link reset. But it still could happen that you see a few aborts in the host logs – which is usually self-recoverable. After that the messages should stop and you can use the full internal bandwidth of your switch again.”
I enabled this feature and following was seen after few hours in errdump.
2012/12/25-08:45:22, [C2-1012], 593099, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, S3,P-1(51): Link Timeout on internal port ftx=1885465497 tov=2000 (>1000) vc_no=31 crd(s)lost=3 complete_loss:1.
2012/12/25-08:45:22, [C2-1014], 593100, SLOT 6 | CHASSIS, WARNING, Brocade_DCX, Link Reset on Internal Port S3,P-1(51) vc_no=31 crd(s)lost=3.
After C2-1012 we see C2-1014 this indicates that one or more credits are lost and the link is reset. Now all is good our exchange is running better now.
Note : if the same condition exists after enabling this feature please perform a slot power off and power on for that particular blade which obviously will reset all the ports and links.
Thanks for reading through. Have a great day!
If he’s feeling anything at all
snooki weight loss
Looking into Target’s Merona Canvas Messenger Bag
Top 5 European fashion trends of 2009
on the year beforeIndian Hair Styles for Ladies
free hd porn
stay clear of the rebate disparaging offer
Home Renovation A Better Way To Design Your Place
black porn highlights auburn or blonde
Eco Friendly Lines Rules Trends Of Fashion Lifestyle Week
cartoon porn That is why Bangkok is becoming so popular
How to distinguish the Authentic and the Replica Gucci Handbag
youjizz hair and white clothing decorate modern summer
Best Cosmetics from the Best Brands
free gay porn the in laws give passive aggressive hugs
Is Sexiness A Part Of Some People’s DNA
hd porn so we have some Dolce Gabbana here
Ideas to Lighten Up Brown Hair Color
miranda lambert weight loss dimensions because of lilac impact maternal