MJ Network Solutions Blog

Dealing with shitty numbers a follow up to NEW STORAGE A-YAY! (Array)

boss — Sun, 20 Jan 2013 08:00:59 +0000

After tallking with others and just realizing how shitty my numbers were when it came to disk/array perfomance testing I decided to get back into bonnie++ testing. I feel much better with these numbers though because I know the power of bonnie along with the simplicity…well this wasnt so simple but it is a bit more automated. Also while running this I saw some very interesting numbers when looking at zpool iostat 1K+ write operations which was nice. These numbers are more realistic and can be used in a conversation to describe performance since it uses a fairly well known tool. It will become my standard along with the usually test of dd’ing zeros…I still like to see those big numbers. :)

Source: http://www.krazyworks.com/testing-filesystem-performance-with-bonnie/

8x146GB 15k with LVM on top using ext4 via Hardware RAID6

bonnie++ -n 0 -u 0 -r `free -m | grep ‘Mem:’ | awk ‘{print $2}’` -s $(echo “scale=0;`free -m | grep ‘Mem:’ | awk ‘{print $2}’`*2″ | bc -l) -f -b -d /var/lib/vz/r6-15k-146-array/
Using uid:0, gid:0.
Writing intelligently…done
Rewriting…done
Reading intelligently…done
start ‘em…done…done…done…done…done…

Version 1.96 ——Sequential Output—— –Sequential Input- –Random- Concurrency 1 -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks– Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
h1 32052M 467993 60 177575 23 415967 25 970.4 42 Latency 326ms 543ms 119ms 408ms

Seq Write: 457MB/s
Seq Re-Write: 173MB/s
Seq Read: 406MB/s

8x1TB 7.2k using ZFS RZ2 (RAID6) with lzjb compression enabled

First run on new array

bonnie++ -n 0 -u 0 -r `free -m | grep ‘Mem:’ | awk ‘{print $2}’` -s $(echo “scale=0;`free -m | grep ‘Mem:’ | awk ‘{print $2}’`*2″ | bc -l) -f -b -d /mnt/datapool2/
Using uid:0, gid:0.
Writing intelligently…done
Rewriting…done
Reading intelligently…done
start ‘em…done…done…done…done…done…
Version 1.96       ——Sequential Output—— –Sequential Input- –Random-
Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

h1           32052M           92609 16 81752 14           568497 31 310.7   3 Latency                         610ms     694ms             57049us     388ms

Seq Write: 90MB/s
Seq Re-Write: 79.8MB/s
Seq Read: 555MB/s

Second run on new array

Using uid:0, gid:0.
Writing intelligently…done
Rewriting…done
Reading intelligently…done
start ‘em…done…done…done…done…done…
Version 1.96       ——Sequential Output—— –Sequential Input- –Random-

Concurrency   1     -Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
h1           32052M           94759 15 81940 14           557672 31 313.9   3
Latency                         613ms     524ms             83193us     339ms

Seq Write: 92.5MB/s
Seq Re-Write: 80MB/s
Seq Read: 544MB/s

During Writing intelligently

zpool iostat datapool2 1
capacity     operations    bandwidth
pool        alloc   free   read write   read write
———- —– —– —– —– —– —–
datapool2   1.99T 5.26T      0      0      0      0
datapool2   1.99T 5.26T      0      0      0      0
datapool2   1.99T 5.26T      0      0      0      0
datapool2   1.99T 5.26T      0      0      0      0
datapool2   1.99T 5.26T      0      0      0 4.00K
datapool2   1.99T 5.26T      0    684      0 2.60M
datapool2   1.99T 5.26T      0 1.15K      0 4.77M
datapool2   1.99T 5.26T      5    633 5.00K 2.46M
datapool2   1.99T 5.26T      0    602      0 2.41M
datapool2   1.99T 5.26T      0 1.15K      0 4.79M
datapool2   1.99T 5.26T      0    654      0 2.48M
datapool2   1.99T 5.26T      0    955      0 3.92M
datapool2   1.99T 5.26T      0    858      0 3.30M

During Rewriting

zpool iostat datapool2 1
capacity     operations    bandwidth
pool        alloc   free   read write   read write
———- —– —– —– —– —– —–
datapool2   1.99T 5.26T    521 1.18K 2.28M 4.84M
datapool2   1.99T 5.26T    778    604 3.41M 2.42M
datapool2   1.99T 5.26T    783    604 3.42M 2.42M
datapool2   1.99T 5.26T    520 1.18K 2.27M 4.85M
datapool2   1.99T 5.26T    640    604 2.80M 2.43M
datapool2   1.99T 5.26T    658    604 2.88M 2.43M
datapool2   1.99T 5.26T    774    633 3.41M 2.55M
datapool2   1.99T 5.26T    524 1.16K 2.28M 4.75M
datapool2   1.99T 5.26T    778    604 3.41M 2.42M
datapool2   1.99T 5.26T    776    604 3.41M 2.43M
datapool2   1.99T 5.26T    525 1.18K 2.28M 4.85M

During Reading intelligently

zpool iostat datapool2 1
capacity     operations    bandwidth
pool        alloc   free   read write   read write
———- —– —– —– —– —– —–
datapool2   1.99T 5.26T 4.06K      0 18.3M      0
datapool2   1.99T 5.26T 4.03K      0 18.2M      0
datapool2   1.99T 5.26T 4.03K      0 18.2M      0
datapool2   1.99T 5.26T 4.10K      0 18.5M      0
datapool2   1.99T 5.26T 4.10K      0 18.5M      0
datapool2   1.99T 5.26T 4.14K      0 18.7M      0
datapool2   1.99T 5.26T 4.03K      0 18.2M      0
datapool2   1.99T 5.26T 4.03K      0 18.2M      0
datapool2   1.99T 5.26T 4.24K      0 19.1M      0
datapool2   1.99T 5.26T 4.07K      0 18.3M      0
datapool2   1.99T 5.26T 4.03K      0 18.2M      0

During start ‘em

zpool iostat datapool2 1
capacity     operations    bandwidth
pool        alloc   free   read write   read write
———- —– —– —– —– —– —–
datapool2   1.99T 5.26T    311     31 1.48M   384K
datapool2   1.99T 5.26T    318     27 1.50M   336K
datapool2   1.99T 5.26T    305     29 1.43M   360K
datapool2   1.99T 5.26T    321     35 1.51M   432K
datapool2   1.99T 5.26T    301     41 1.43M   504K
datapool2   1.99T 5.26T    329     31 1.55M   384K
datapool2   1.99T 5.26T    308     20 1.46M   252K
datapool2   1.99T 5.26T    316     36 1.49M   444K
datapool2   1.99T 5.26T    326     28 1.55M   348K

Transferring ISO between arrays using rsync

FROM 8x1TB 7.2k (ZFS RZ2)
TO 8x146GB 15k (Hardware RAID6 w/ LVM on top)

rsync -a –progress /mnt/datapool2/vm/template/iso /var/lib/vz/r6-15k-146-array/test/
sending incremental file list
created directory /var/lib/vz/r6-15k-146-array/test
iso/
iso/FreePBX-1.1007.210.58-x86_64-Full-1344904533.iso
827643904 100% 131.17MB/s    0:00:06 (xfer#1, to-check=14/16)
iso/FreePBX-1.815.210.58-x86_64-Full-1344903580.iso
827457536 100% 135.75MB/s    0:00:05 (xfer#2, to-check=13/16)
iso/Windows-7-Home-Premium-x64.iso
3731709952 100% 130.39MB/s    0:00:27 (xfer#3, to-check=12/16)
iso/Windows-XP-SP3-Automated.iso
648937472 100% 130.04MB/s    0:00:04 (xfer#4, to-check=11/16)
iso/clearos-community-6.3.0-x86_64.iso
663922688 100% 117.69MB/s    0:00:05 (xfer#5, to-check=10/16)
iso/oi-dev-151a-text-x86.iso

…..

sent 12959018903 bytes received 301 bytes 141628625.18 bytes/sec
total size is 12957435904 speedup is 1.00

TO 8x1TB 7.2k (ZFS RZ2)
FROM 8x146GB 15k (Hardware RAID6 w/ LVM on top)

rsync -a –progress /var/lib/vz/r6-15k-146-array/test/ /mnt/datapool2/
sending incremental file list
./
iso/
iso/FreePBX-1.1007.210.58-x86_64-Full-1344904533.iso
827643904 100%   78.86MB/s    0:00:10 (xfer#1, to-check=14/17)
iso/FreePBX-1.815.210.58-x86_64-Full-1344903580.iso
827457536 100%   78.87MB/s    0:00:10 (xfer#2, to-check=13/17)
iso/Windows-7-Home-Premium-x64.iso
3731709952 100%   80.21MB/s    0:00:44 (xfer#3, to-check=12/17)
iso/Windows-XP-SP3-Automated.iso
648937472 100%   68.44MB/s    0:00:09 (xfer#4, to-check=11/17)
iso/clearos-community-6.3.0-x86_64.iso
663922688 100%   77.94MB/s    0:00:08 (xfer#5, to-check=10/17)
iso/oi-dev-151a-text-x86.iso
514420736 100%   75.98MB/s    0:00:06 (xfer#6, to-check=9/17)

NEW STORAGE A-YAY! (Array)

boss — Fri, 18 Jan 2013 15:07:12 +0000

New array built of SAS/SATA JBOD unit w/ 8x 1TB Seagate Constelattion ES drives connected to an Adaptec 6445 card in RZ2 (RAID6)

root@h1:~# dd if=/dev/zero of=/mnt/datapool2/test000 bs=1G count=4

4+0 records in

4+0 records out

4294967296 bytes (4.3 GB) copied, 28.622 s, 150 MB/s

Old array built of Dell 220S DAS units w/ 30x 146GB 10k U320 drives in RZ2 (RAID6)

This has Compression is enabled

root@h1:~# dd if=/dev/zero of=/mnt/datapool1/test000 bs=1G count=4

4+0 records in

4+0 records out

4294967296 bytes (4.3 GB) copied, 55.3248 s, 77.6 MB/s

So to be fair or make sure everything is the same I enabled compression on the new array as well.

root@h1:~# zfs set compression=lzjb datapool2

root@h1:~# dd if=/dev/zero of=/mnt/datapool2/test000 bs=1G count=4

4+0 records in

4+0 records out

4294967296 bytes (4.3 GB) copied, 16.6144 s, 259 MB/s

A bit unbelievable so lets make that file twice the size.

root@h1:~# dd if=/dev/zero of=/mnt/datapool2/test000 bs=1G count=8

8+0 records in

8+0 records out

8589934592 bytes (8.6 GB) copied, 32.7574 s, 262 MB/s

THATS CAPTAIN FUCKING INSANO SON!

Spent about 1,400 on this project to get rid of the power hungry Dell 220S units and gain more storage space. Splurged on the drives by going with the higher end SATA enterprise series (Seagate constellation ES). These drives and the card can perform faster at 6GB/s but the SAS Enclosure is 3GB/s which is still roughly 9.3 times faster than old school U320 SCSI drives. If it were 6GB/s end to end I would just have to change my pants and start wearing diapers every time I transfer files. Plus I havent even added the SSD drives as ZIL device which will speed things up even more!

PICs Coming soon.

Automatic Tape Labeling for Bacula

boss — Sun, 05 Aug 2012 09:30:50 +0000

I have been working with Bacula recently on a new project to help fill a void for our customers. Namely the lack of back up solution for their data. So we are putting something together…more info coming soon. So we got Bacula configured and made it more automated to do simple things like mounting/un-mounting|ejecting the tape. But no matter how much I research I cant find out how to have Bacula label the tape automatically. Maybe I did something wrong but the options just dont work for me the way I want it to. So using a mount/umount script I found earlier on the bacula manual I added the label command then used the date command to label the media for the day the backup is ran.

My setup is simple here, there are three tapes labeled Monday, Wednesday, and Friday in a single pool called MWFPool. The job runs the script to mount a drive before hand and on the last job in the queue it unmounts the drive and issues an mt command to (offline) rewind and eject the tape. Like bread from the toaster. Signaling that the drive needs a new tape put in. Took a second to get right since my programming/scripting is weak but I got it to work successfully.

/etc/bacula/scripts/bacula-mount
#!/bin/bash
#This gets the Day of the week and maps it to the variable.
day_of_week=$(/bin/date +”%A”)

#This is done to mount the tape drive for backups.User must of course push in the tape. Label command added with all the options

/usr/bin/bconsole -c /etc/bacula/bconsole.conf < mount storage=LTO2Drive
label storage=LTO2Drive volume=”$day_of_week” pool=MWFPool
END_OF_DATA

Really simple right!

/etc/bacula/scripts/bacula-umount
#!/bin/bash

#This is done to un mount the tape drive and eject it from the system.

/usr/bin/bconsole -c /etc/bacula/bconsole.conf < unmount storage=LTO2Drive
END_OF_DATA

#This will eject the tape from the drive for the user to take.
/bin/mt -f /dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039 offline

This makes it easier for customers to add new tapes if they want without me having to log in and run the command. The less we have to do of the smaller things the better for both parties. Change the device name however you need to for your setup.

I just like using the /dev/tape/by-id/ method more.

Output of each command

/etc/bacula/scripts/bacula-mount Connecting to Director localhost:9101 1000 OK: mjnsbackupbox0-dir Version: 5.2.5 (26 January 2012) Enter a period to cancel a command. mount storage=LTO2Drive Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" 3905 Device ""LTO2" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039)" open but no Bacula volume is mounted. If this is not a blank tape, try unmounting and remounting the Volume. You have messages. label storage=LTO2Drive volume="Sunday" pool=MWFPool Connecting to Storage daemon LTO2Drive at 127.0.0.1:9103 ... Sending label command for Volume "Sunday" Slot 0 ... block.c:1001 Read error on fd=5 at file:blk 0:0 on device "LTO2" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039). ERR=Input/output error. 3000 OK label. VolBytes=1846204416 DVD=0 Volume="Sunday" Device="LTO2" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039) Catalog record for Volume "Sunday", Slot 0 successfully created. Requesting to mount LTO2 ... 3001 Device ""LTO2" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039)" is mounted with Volume "Sunday"

Rerun it

/etc/bacula/scripts/bacula-mount Connecting to Director localhost:9101 1000 OK: mjnsbackupbox0-dir Version: 5.2.5 (26 January 2012) Enter a period to cancel a command. mount storage=LTO2Drive Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" 3001 Device ""LTO2" (/dev/tape/by-id/scsi-1IBM_ULTRIUM-TD2_1110247039)" is mounted with Volume "Sunday" You have messages. label storage=LTO2Drive volume="Sunday" pool=MWFPool Media record for new Volume "Sunday" already exists.

Proxmox 2.1 – Installing ZFS

boss — Thu, 02 Aug 2012 13:35:55 +0000

I document quite a bit of tutorials and self made howtos in our intranet site so I decided to share this with everyone. It is mainly from the Proxmox wiki but the last few parts are from around the net and my knowledge. As you can see in the Proxmox wiki it added somethings later that fixed my problem but I had to reinstall the packages from scratch.

By the way

dpkg –get-selections | grep
dokg –purge
is a great time saver when trying to fix what mistakes were made.

Feel free to email us if there is something wrong or not necessary.

Source:

http://pve.proxmox.com/wiki/ZFS#Native_ZFS_for_Linux_on_Proxmox_2.0

Basically watch what kernel headers you have and make a symbolic link that allows certain packages to build properly,

Install instructions:
apt-key adv –keyserver keyserver.ubuntu.com –recv-keys F6B0FC61
aptitude update

Be warry of which kernel you are running with this one. Change as necessary.

ln -s /lib/modules/2.6.32-11-pve/build /lib/modules/2.6.32-11-pve/source
aptitude install pve-headers-2.6.32-11-pve
aptitude install dkms pve-headers-$(uname -r)
aptitude install ubuntu-zfs

I didn’t seem to need this one but I do it anyway since Im paranoid.
aptitude install spl-dkms

Just in case the kernel gets upgraded do this
aptitude install dkms pve-headers-$(uname -r)
then this to rebuild the packages
aptitude reinstall spl-dkms zfs-dkms

Now lets get it to startup with the system as well. Add the following to /etc/rc.local

#Mount ZFS storage
/usr/local/sbin/zfs-fuse
/usr/local/sbin/zfs mount -a

When the system reboots it tends to mount the directory that zfs has set as a mountpoint before zfs gets a chance since the zfs mount command is called in /etc/rc.local. I want to keep things as pristine as possible so this all can work even if an update occurs that changes things a bit so I will keep it as it is. The only thing I can think of is to make the commands apart of an /etc/init.d script then place it above file system mounting better yet above what ever proxmox is running to cause this.
I believe the issue is from what we set in the “Storage” tab. So i left that the same and will not bother it. Ran the following which creates a symlink to zfs new mountpoint which i changed with the command below.

Change zfs mount point directory no need to create it beforehand.
zfs set mountpoint /mnt/datapool0/vm datapool0

Mount it
zfs mount -a

Then run this to remove the vm subdirectory and create a link to the correct place
rm -rf /datapool0/vm;ln -s /mnt/datapool0/vm /datapool0/vm

Reboot and test. This was a quick fix and still work after reboots. Essentially just create some BS directory then symlink it to the real one afterward.

***UPDATE 08/05/2012***
I had an issue with the backups using snapshot instead of suspend. Suspend causes down time so I researched the error.

Undefined subroutine &PVE::Storage::cluster_lock_storage called at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 240

Found this post:
http://forum.proxmox.com/archive/index.php/t-10438.html

I feared that this may mess up the current install of zfs and its dependencies being that we needed the kernel headers before. So I ran the suggested fix and it all came up after a reboot.

“aptitude update && aptitude full-upgrade” then reboot
The backups are working as expected using snapshot and not suspend.

Fuuuucckkkk a Cisco Access server…

boss — Mon, 23 Apr 2012 04:14:26 +0000

Yes i know this is a company blog and it represents who we are as collective, but have you seen the outrageous price of a Cisco access server the Cisco 2511. Its kinda ridiculous and since we are constructing a lab for certification purposes we don’t feel like spending $180 plus on a device that will allow us to log into the console port of our routers/switches. So we thought why not build our own Cisco Access Server. We don’t need anything fancy just some computer with a lot of Serial ports. LOL…I had to laugh at that last line, “We don’t need anything fancy…” cause we used a Dell 2950 that has 8GB of Ram, 2x Quad core Xeon processors with the standard RAID 1 setup of 250GB drives, buuuutttt you can do it with whatever. This PC is running Ubuntu Desktop so we can log into it anytime via VNC to play with the setup. This allows us to leverage an 64 bit OS that can have packages installed like:

TFTP – For conf and IOS backup practice
NMAP – To test ACLs/Port Forwards
VBox – Why simulate the functions of a Windows box when you can use it in a VM if you REALLY want to
Media Access – We have tons of tutorials, videos, and other things on the NAS at our disposal that can be access from this box.
Putty/Minicom – Allows us to access the devices via the serial ports either with a gui/cli. Putty is just more popular but there are others.
Packet Tracer – I have to find a 64bit version of this or we can stick it in a VM

So we found a 6 port Serial PCI card on eBay for $28 (YEAH BABY!). It was the SD-PCI-6S by Syba. There were cheaper sellers but I didn’t feel like waiting a month for shipping from overseas. It arrived in no time and installation didn’t require anything in the BIOS just a minor software tweak which I got figured out with Ubuntu.

I had to some trouble with all the PCI Card’s serial ports being recognized and mapped to a /dev file. I was able to access the first 2 ports on the card and remap any port to the special ttySx (where ‘x’ represents a number) file. It turned out to be something that had to be added to grub conf file like so:

nano /etc/default/grub

Find the line:
GRUB_CMDLINE_LINUX_DEFAULT=”" <—may be blank between quotes if not add a space between arguments

change it to:
GRUB_CMDLINE_LINUX_DEFAULT=”8250.nr_uarts=8″ <—- 8 represents the number of ports you may have. I added 6 so 8 was correct

commit the change by rebuilding the grub menu with the new arguments in place so it survives reboots:
sudo update-grub

Taken from “Beyond Bits and Bytes” Blog

Now all ports work after reboot. I added labels to the ports on the back of the server. Then got VNC to run when the machine starts which is essential for remote lab time. This will be great for us and our friends who want some lab time for certifications. Smarter fiends is always a plus.

As far as cost savings this was a very good move if I must say so myself. We get all the ports we need for the lab plus the added functionality of a full blown server that is over spec’d for this project. Granted the Dell 2950 is overkill, but it was going to be used for development purposes anyway. So $28 bucks for a PCI card versus $180+ for a console port switcher…SO WORTH IT. Also this card can be used in a desktop box as well with no issue and the extra money saved went to buying another 3550-24 PWR to complete the lab.

Ahh yes its coming together nicely

boss — Mon, 02 Apr 2012 06:00:26 +0000

With Jumbo Frames enabled on the new gig switch, the storage server configured, and the host machines clustered the dev rack is moving towards stability. Now that were almost done with testing its time to get the cabling of this bad boy a little neater. Now with this being a dev rack and all testing is never truly over and improvements will always be made but it will look nice doing it from now on. Were getting great numbers from the NFS shares concerning transfer speeds without any tweaks. Running a quick back in Proxmox 2.0 i get 87 Mbps from yellow array (RAIDZ 146GB 15K SAS) which is currently degraded (replacement drive in route) to the orange array (RAIDZ2 146GB 10K U320 SCSI w/ gzip compression enabled). So far so good but the next project is some sort of off site tape backup for the data. FreeNas is great with its daily logs and prompt warning when finding issues. More to come…

Next plan of action

boss — Thu, 29 Mar 2012 07:46:24 +0000

We have been working with our customers a lot lately which is good for us. Some are requiring more servers and we are happy to oblige them of course.

So far FreeNAS is our distro of choice for storage systems on the Dell Power Edge 840 systems. Hot swap is nice if you can get it. A low end model with a Xeon dual core proc, 1GB of RAM, no hdd, no hot swap start at $100 with shipping. This is great for small offices. Then all that is needed is an 8GB flash drive for ~$10 off newegg and some HDDs of your choice. Using ZFS to RAID the drives from their. Compression enabled makes the disk space last longer as well. Which is a plus for small businesses wanting to get the most from their storage. We are going to start working with the RD1000 for take home backups of the most important data on the storage server as well.

Next is more monitoring and log shipping for or user’s systems. We will offer it as a free service for simple monitoring like logs to emails and interface outages on important ports. With Cacti and an SMTP relay this isn’t hard to do we just need the back end built out for it. Which is the reason for the dev racks VM servers.

Our documentation skills have improved as well. We document with detail and it has helped us out a great deal when troubleshooting. So we use Open Atrium with a group for each customer to help us keep the info seperate and to allow for customer interaction later if we choose.

So were hanging in there and making improvements to the company, our services, and ourselves as well. PEACE, LOVE, and HIGH SPEED DATA TRANSFERS

JUMBO FRAMES

boss — Thu, 29 Mar 2012 07:25:24 +0000

So a new friend of mine explained Jumbo frames to me and I have heard about it before but his explanation was super simple. It goes a little something like this.

You got 1 packets with an MTU of 1500 a piece which is the default for Fast Ethernet (100Mbps). A jumbo frame is really anything larger than this value for Gigabit Ethernet (1000Mbps) switches the common maximum is 9000 with some higher end switches supporting larger sizes. So with CPU, TCP/IP, and wire speed over head transferring those 6 packets takes time and resources. Of course this overhead and processing is miniscual in most environments. Now with jumbo frames on gigabit interfaces with an MTU of 9000 those 6 packets are now just one big packet. You can say that’s 1/6 the CPU, TCP/IP, and wire speed resources it takes to transfer that same amount of information. Of course real world calculations were not done on that simply because i don’t feel like figuring it out, but you get the point. So the hosts are working less but transferring the same payload and the switch is working less as well because it has less packets to push/process.

To sum it up you ship 1 big elephant instead of 6 baby elephants (dam that was pretty good)…but your like me and saying so what will my data transfer speeds be higher? I soon found that they actually are my friend.

Background/Environment:

- Cisco 2970G-24T coming in at a whopping $15 per port…oooo so worth it!
- Freenas with 2xGig ports with LACP and Jumbo frames enabled on the interface and switch ports connected to a RAIDZ2 pool of 14 146GB 10K u320 SCSI drives. Known as “orange”. Dam right I color code my arrays!

- Sharing Orange via NFS

- 2x Proxmox 2.0 Beta systems with 2xGig ports with LACP and Jumbo frames enabled on the interface and switch ports mounted to orange via NFS.

This took a bit of time to get right and i will include confs and links to that I used to get it done.

host01: change mtu/confirm mtu/test writing a 4gig file to the NFS share

root@host01:~# ifconfig bond0.3 mtu 1500
root@host01:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.15
cache mtu 1500 advmss 1460 hoplimit 64
root@host01:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0001 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 50.1927 s, 85.6 MB/s
root@host01:~# ifconfig bond0.3 mtu 9000
root@host01:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.15
cache mtu 9000 advmss 8960 hoplimit 64
root@host01:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0001 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 38.5115 s, 112 MB/s

host02: change mtu/confirm mtu/test writing a 4gig file to the NFS share

root@host02:~# ifconfig bond0.3 mtu 1500
root@host02:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.16
cache mtu 1500 advmss 1460 hoplimit 64
root@host02:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0002 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 62.7919 s, 68.4 MB/s
root@host02:~# ifconfig bond0.3 mtu 9000
root@host02:~# ip route get 172.18.3.9
172.18.3.9 dev vmbr3 src 172.18.3.16
cache mtu 9000 advmss 8960 hoplimit 64
root@host02:~# dd if=/dev/zero of=/mnt/pve/vm-orange/test0002 bs=1G count=4
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB) copied, 57.3596 s, 74.9 MB/s

Not sure yet why I get different speeds from the boxes but I assume it may have to do with the amount of memory in each system. Host01 has 8GB while Host02 4GB. I will deal with that in the future though $$$.

You see that after the change we have an increase in transfer speeds so the benefits are apparent and with faster arrays, more memory, and the possible improvements to 10Gbps interface (I FUCKING WISH) we can squeeze even more out of this. Also don’t forget about tweaks to NFS and SMB, this can really give you improvements without the expensive upgrades.

Confs and articles:

Cisco 2970 – Enable jumbo frames on all gigabit interfaces. This requires a reload to take affect.
(config)#system mtu jumbo 9000

FreeNAS-8.0.3-RELEASE-p1-x64 (9591) Documentation (read it like a bible) Video (older version but good detail)
Go to Network > Link Aggregation > View Link Aggregations > Edit Interface > In options field enter “mtu 9000″ > save and reboot to make sure it stuck.

Proxmox 2.0 beta – This allows for the mtu change to survive reboots. This is the same for debian/ubuntu.
Edit the /etc/rc.local file with the following format: ifconfig mtu 9000
Ex:
ifconfig eth0 mtu 9000
ifconfig eth1 mtu 9000
ifconfig bond0 mtu 9000
ifconfig bond0.3 mtu 9000

So I think I found the problem…

boss — Thu, 16 Feb 2012 09:31:46 +0000

This storage server has been a back and forth ordeal and I was kicking my self profusely trying to find what the hell has gone awry here. Well after spending more money :( getting another HBA for the SCSI enclosure and finding that the card doesn’t work for my purpose, which was a serious downer. I just ordered the PCI-X riser for the 2950 so i can use some of the old cards I have lying around. In the mean time i started getting errors about memory and the CPU from the LCD on the front…WTF!!!! I just got these Procs (at a bargain) and one is bad. Sorry to say i got them from two different vendors so the packing is mixed up and scattered around my work space. It was confirmed when i used memtest and the box froze completely after the first pass. I took out one proc and reran it then left it. I will check on it later, but that’s crazy.

This might explain the I/O Whoas issue with the storage array. Meaning that the first RAID card may not have been an issue after all. Hmmmm…DAMMIT!!! Well i shall move forward and replace the proc after I run a multi pass memtest to make sure the current proc is still good.

UPDATE:

Well a little after I wrote the first portion of this I found that the 8GB of mem that I had may have been the issue but even after testing further I got an error that shows that one of the DIMM slots may have been an issue. So I reseated everything and tested again after deciding to re-purpose the box and oddly enough it passed 4 tests no issues in mem test nor on the Dell LCD. O well I don’t care to make this a storage server any more so it will do as is and since it passed the memtest i guess its good for a dev box. :)

When will this ever be finished….

boss — Thu, 16 Feb 2012 09:21:02 +0000

NEVER!! As long as i keep learning something new i will continually update and change the configuration of this rack. I have found myself asking that question and finally i understand why its never finished. Im always finding new projects and as a company we are continually tested by our customers.

So this will never be done in a sense that nothing will be added but some day it wont be changed ever again…like when im dead for instance :). So with this current change I am adding two Sunfire x4100 M2 servers. Converting our old production VM server to a storage server using FreeNAS w/ ZFS, Adaptec U320 SCSI and 5 series SAS controllers, a Chenbro SAS Expander that will be housed in a Supermicro case later on with a JBOD board. Then I will use the Dell 2950 as test bed for my new venture into programming/scripting.

I finally stopped fighting it and said FUCK IT, DO IT LIVE! its time to roll with the big boys. So that box will be my learning platform for Open Solaris and ruby shell scripting. Im hoping to create a search engine like Google’s Search Appliance that will index all the information across the various systems we use to run our business. From File shares to web based information. Should be easy right? Well im gonna take baby steps and do a few Hello World programs first.

The Sunfire boxes are awesome with the iLOM boards that allow me to do remote power functions and even watch the server boot from BIOS as if a screen was directly attached to it OUT OF THE BOX! I know a lot of exclamation points but im excited. This will be a great learning platform for us and we will build on it as always.

Time for show an tell

(click on pics for a bit of a description)