One thing I noticed is that every single installation I have seen of iSCSI in VMware there has been crucial items missed in the setup that always rears its ugly head at the most inopportune time. I have poured over numerous blogs and best practices and it all becomes very dizzying. Heck throw in the fact that EVERY single manufacture has their own Best Practices it is no wonder people can’t get it right. I will attempt to document it in a step by step process and also highlight key things to keep in mind when building out iSCSI networks.
First and foremost before you even start on setting up iSCSI you really need to see what kind of networking you will need. Selecting the right networking switches is CRUCIAL to your overall success of an iSCSI implementation. Want to go with the SMB switches that is fine but beware if you are pushing any kind of I/O through it, it’s probably going to drop packets like a banshee. So let’s take a closer look at Cisco 2960 switches for argument’s sake we will use a Cisco 2960 -48 port switch.
Each portion of the switch highlighted above goes to a specific port Asic’s.
Putty into your switch and go into enable mode once there type in this: sh platform port-asic version (this is a output from a 3750 just to give you a visual)
This will then show you how many port asic you have in your switch. Typically for Cisco on a 48 port switch they break it out into the first 24 ports go to one port Asic (red box above) second set of 24 ports go to another port asic ( greenbox above) then finally the uplink ports are dedicated their own port asic (indicated by yellow box above). Again another way to tell this would be to run this command from your putty session: show platform pm if-numbers pay special attention to the column labeled Port as this will correlate to the Asic the corresponding interface belongs to.
So you’re probably wondering ok I thought this was a iSCSI post trust me it is but we really need to make sure we have our foundation solid before we build things out. Two final things when selecting your switch you need to make sure that your switch is able to handle bursty traffic part of being able to do this is having ample port buffers and whether those port buffers are shared or dedicated makes a huge difference. For instance on the 2960’s they have a 384K shared port buffer that is shared between 12 ports. So if you are going to be doing any sort of high traffic and virtualization through a 2960 just be cognizant that you are going to be walking a fine line depending on how many host you have and what the VM’s are doing. So one way to combat this is not to load up one port asic with all the iSCSI traffic. So ideally you would have at least 2 dedicated switches for iSCSI traffic only.
So let’s make a realistic scenario 3 ESXi host, VNXe and a pair of dedicated 2960 for iSCSI and another pair of 2960’s for production traffic.
Host 1
Host 2
Host 3
iSCSI switches
Infra-ISCSI 1
Infra-ISCSI 2
Prod switches
Infra- Prod1
Infra- Prod2
Here is what the cabling would look like for the three host
Host1 to Infra-ISCSI1 put 1 Ethernet port from Host1 into Gi1/1
Put the 1 Ethernet port from Host1 into Infra-ISCSI2 into Gi1/1
2 ports from Host 1 into Infra-Prod1 Gi1/1 and Gi1/2
Put 2 ports from Host 1 into Infra-Prod2 Gi1/1 and Gi1/2
Host2 to Infra-ISCSI1 put 1 Ethernet port from Host 2 into Gi1/25
Put the 1 Ethernet port from Host2 into Infra-ISCSI2 into Gi1/25
2 ports from Host2 into Infra-Prod1 Gi1/25 and Gi1/26
Put 2 ports from Host2 into Infra-Prod 2 Gi1/25 and Gi1/26
Host3 to Infra-ISCSI1 put 1 Ethernet port from Host3 into Gi1/2
Put the 1 Ethernet port from Host3 into Infra-ISCSI2 into Gi1/2
2 ports from Host 1 into Infra-Prod1 Gi1/3 and Gi1/4
Put 2 ports from Host 1 into Infra-Prod2 Gi1/3 and Gi1/4
VNXE
SPA Mgmt 1 port going to Infra-Prod1 Gi1/48
SPA iSCSI ports-1 port to Infra-ISCSI1 Gi1/48 and 1 port to Infra-ISCSI2 Gi1/24
SPB Mgmt 1 port going to Infra-Prod1 Gi1/48
SPB iSCSI ports- 1 port to Infra-iSCSI1 GI1/24 and 1 port to Infra-ISCSI2 Gi1/48
The above setup ensures that the ports are spread out evenly across the port asic and set you up for expansion with a blueprint of sorts already laid out. You might be looking at the host and going that is only 6 ports. So for ESXi there would be 2 for Mgmt/vMotion, 2 for Production Traffic and 2 for dedicated iSCSI traffic. If you just so happened to have 8 NICS per server then I would move vMotion to its own NIC’s. That is a lot of copper though if you are going to have 4 ESXi host that is 32 ports so you can eat up a ton of ports pretty quickly.
I used 2960’s for this demonstration but you may really want to look at Switches with much larger port buffers like 3750 or the 4850. If you want to get into the Nexus line on Cisco then 3K’s is where you would enter in. So now that we have our networking setup then we can move onto Part 2… I mean we have to get this right so we can use our new virtual VPLEX right?
Diagram of Above Port layout
Girard's Corner 