We have been working with our customers a lot lately which is good for us. Some are requiring more servers and we are happy to oblige them of course.

So far FreeNAS is our distro of choice for storage systems on the Dell Power Edge 840 systems. Hot swap is nice if you can get it. A low end model with a Xeon dual core proc, 1GB of RAM, no hdd, no hot swap start at $100 with shipping. This is great for small offices. Then all that is needed is an 8GB flash drive for ~$10  off newegg and some HDDs of your choice. Using ZFS to RAID the drives from their. Compression enabled makes the disk space last longer as well. Which is a plus for small businesses wanting to get the most from their storage. We are going to start working with the RD1000 for take home backups of the most important data on the storage server as well.

Next is more monitoring and log shipping for or user’s systems. We will offer it as a free service for simple monitoring like logs to emails and interface outages on important ports. With Cacti and an SMTP relay this isn’t hard to do we just need the back end built out for it. Which is the reason for the dev racks VM servers.

Our documentation skills have improved as well. We document with detail and it has helped us out a great deal when troubleshooting. So we use Open Atrium with a group for each customer to help us keep the info seperate and to allow for customer interaction later if we choose.

So were hanging in there and making improvements to the company, our services, and ourselves as well. PEACE, LOVE, and HIGH SPEED DATA TRANSFERS

So a new friend of mine explained Jumbo frames to me and I have heard about it before but his explanation was super simple. It goes a little something like this.

You got 1 packets with an MTU of 1500 a piece which is the default for Fast Ethernet (100Mbps). A jumbo frame is really anything larger than this value for Gigabit Ethernet (1000Mbps) switches the common maximum is 9000 with some higher end switches supporting larger sizes. So with CPU, TCP/IP, and wire speed over head transferring those 6 packets takes time and resources. Of course this overhead and processing is miniscual in most environments. Now with jumbo frames on gigabit interfaces with an MTU of 9000 those 6 packets are now just one big packet. You can say that’s 1/6 the CPU, TCP/IP, and wire speed resources it takes to transfer that same amount of information. Of course real world calculations were not done on that simply because i don’t feel like figuring it out, but you get the point. So the hosts are working less but transferring the same payload and the switch is working less as well because it has less packets to push/process.

To sum it up you ship 1 big elephant instead of 6 baby elephants (dam that was pretty good)…but your like me and saying so what will my data transfer speeds be higher? I soon found that they actually are my friend.

Background/Environment:

- Cisco 2970G-24T coming in at a whopping $15 per port…oooo so worth it!
- Freenas with 2xGig ports with LACP and Jumbo frames enabled on the interface and switch ports connected to a RAIDZ2 pool of 14 146GB 10K u320 SCSI drives. Known as “orange”. Dam right I color code my arrays!

- Sharing Orange via NFS

- 2x Proxmox 2.0 Beta systems with 2xGig ports with LACP  and Jumbo frames enabled on the interface and switch ports mounted to orange via NFS.

This took a bit of time to get right and i will include confs and links to that I used to get it done. 

host01: change mtu/confirm mtu/test writing a 4gig file to the NFS share

root@host01:~# ifconfig bond0.3 mtu 1500
root@host01:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.15
cache mtu 1500 advmss 1460 hoplimit 64
root@host01:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0001 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 50.1927 s, 85.6 MB/s
root@host01:~# ifconfig bond0.3 mtu 9000
root@host01:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.15
cache mtu 9000 advmss 8960 hoplimit 64
root@host01:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0001 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 38.5115 s, 112 MB/s

host02: change mtu/confirm mtu/test writing a 4gig file to the NFS share

root@host02:~# ifconfig bond0.3 mtu 1500
root@host02:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.16
cache mtu 1500 advmss 1460 hoplimit 64
root@host02:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0002 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 62.7919 s, 68.4 MB/s
root@host02:~# ifconfig bond0.3 mtu 9000
root@host02:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.16
cache mtu 9000 advmss 8960 hoplimit 64
root@host02:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0002 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 57.3596 s, 74.9 MB/s

Not sure yet why I get different speeds from the boxes but I assume it may have to do with the amount of memory in each system. Host01 has 8GB while Host02 4GB. I will deal with that in the future though $$$.

You see that after the change we have an increase in transfer speeds so the benefits are apparent and with faster arrays, more memory, and the possible improvements to 10Gbps interface (I FUCKING WISH) we can squeeze even more out of this. Also don’t forget about tweaks to NFS and SMB, this can really give you improvements without the expensive upgrades.

Confs and articles:

Cisco 2970 – Enable jumbo frames on all gigabit interfaces. This requires a reload to take affect.
(config)#system mtu jumbo 9000

FreeNAS-8.0.3-RELEASE-p1-x64 (9591) Documentation (read it like a bible) Video (older version but good detail)
Go to Network > Link Aggregation >  View Link Aggregations > Edit Interface > In options field enter “mtu 9000″ > save and reboot to make sure it stuck.

Proxmox 2.0 beta – This allows for the mtu change to survive reboots. This is the same for debian/ubuntu.
Edit the  /etc/rc.local file with the following format: ifconfig <interface name physical and or vlan> mtu 9000
Ex:
ifconfig eth0 mtu 9000
ifconfig eth1 mtu 9000
ifconfig bond0 mtu 9000
ifconfig bond0.3 mtu 9000