Go Back   Hardware Canucks > SOFTWARE > O/S's, Drivers & General Software

    
Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old October 24, 2013, 10:04 PM
lowfat's Avatar
Moderator
 
Join Date: Feb 2007
Location: Grande Prairie, AB
Posts: 7,955

My System Specs

Default Adventures w/ an ESXi server @ home.

Just thought I'd start this thread as I n00b my way through an ESXi Hypervisor build and setup. Some of the VM's I plan to run are Pfsense, FreeNAS, some sort of Windows (8/2008/2012?), and some sort of linux (Ubuntu/Fedora?). Possibly more to come down the road as I think of cool stuff.

Hardware wise is as follows

Opteron 6168. 12 cores @ 1.9GHz. Thanks goes to 10e for this recent upgrade.
Supermicro H8SGL-F.
64GB of 1333MHz DDC ECC . Thanks goes to Soullessone21 for the rocking good deal on these.
IBM M1015 Host Bus Adapter (HBA). Flashed this to an LSI 9211-8i-IT bios when I first bought it. I did this to get rid of the RAID and MPTSAS bioses. This way the card doesn't initialize @ bootup and so the drives can be used in passthrough to the OS. IBM ServeRAID M1015 Part 4: Cross flashing to a LSI9211-8i in IT or IR modeServeTheHome ? Server and Workstation Reviews
Intel 80GB X25-M G2.
Intel 30GB X25-E.
Whole bunch of HDDs.

Watercooled on my temporary ghetto bench.

First up I installed ESXI 5.5. Free version is pretty awesome as far as I can tell. https://my.vmware.com/web/vmware/eva...xi5&lp=default
Install went forward enough. Installed it to my X25-M and it took 1-2GB or so.

I then installed the VMWare vSphere Client on my workstation. Same download link as above.

I then went and set the IBM HBA to passthrough. This allows for a guest VM to take direct control over it. This requires a board and CPU to fully support VD-T. Unless you have a workstation or server board, there is a good chance this won't work.


First up is a FreeNAS VM. I had lots of problems w/ Drive Bender in Windows so I decided to move to this. I tried it out earlier this year for a few days but at the time it wasn't exactly what I wanted. Used this guide for what to use for settings. The reason I am using so much ram is ZFS can use a lot. It caches reads to the ram. Plus I have 64GB of ram so why not?

FreeNASŪ in a Virtual Environment - Freenas



I then added the IBM HBA. This is pretty much a requirement for a FreeNAS that has a large number of disks. This allows FreeNAS to use the HBA and all the drives attached to it directly. Otherwise you could only mount virtual disks.


Then I created a ZFS RAIDZ 'array'. Used 4 x 3TB drives plus the 30GB X25-E as a ZIL drive. The ZIL drive is a cache use for synchronous writes to the array. Not sure if it will be beneficial or not to me but I had the drive so I thought minus whale.


I turned on the CIFS and iSCSI services. Created a CIFS share.


Created a guest user. Then went and gave the ZFS volume permissions for guest to read/write to the volume.




I can now access the ZFS RAIDZ storage pool I made from my Windows machines.


The cool thing about FreeNAS is that you can install little apps. They install inside of what they call a jail. Which is a small self contained OS/VM. If the app ever screws up it is completely isolated from the FreeNAS install. Obviously I will skip over some of the stuff here for obvious reasons.


Will stop here for tonight as it is getting late.

Last edited by lowfat; October 24, 2013 at 11:06 PM.
Reply With Quote
  #2 (permalink)  
Old October 24, 2013, 10:29 PM
Shadowmeph's Avatar
Hall Of Fame
F@H
 
Join Date: Oct 2007
Posts: 3,353

My System Specs

Default

Cool I would love to try this SO I am going to be watching this thread :)
Reply With Quote
  #3 (permalink)  
Old October 25, 2013, 05:50 AM
Hall Of Fame
F@H
 
Join Date: Aug 2007
Location: Toronto, ON
Posts: 1,013

My System Specs

Default

Thanks this is the project I'm also trying to gear up for, so I will be following this thread closely :)
__________________
heatware
Reply With Quote
  #4 (permalink)  
Old October 25, 2013, 06:43 AM
Entilza's Avatar
Top Prospect
 
Join Date: Jun 2011
Posts: 105

My System Specs

Default

Enjoy your esxi experiments.

Just FYI, there are many basic intel processors that offer Vt-D passthrough not just server processors.

For example the i5-3570 offers it but the i5-3570K does not. Similarly with the 4670 and 4670k etc. Intel.com can show you which processor supports. Even my 5 year old E8400 has it.

Also you can use any basic machine for esxi testing if anyone is considering it.

Finally, you can also use your ubuntu server as a file server instead of freenas.

Keep posting on what yer up to!
Reply With Quote
  #5 (permalink)  
Old October 25, 2013, 07:33 AM
lowfat's Avatar
Moderator
 
Join Date: Feb 2007
Location: Grande Prairie, AB
Posts: 7,955

My System Specs

Default

From what I've read a lot of boards aren't designed properly to pass through pcie slots. Boards that share pcie bandwidth between multiple slots or devices can be an issue.

Setting up ZFS / RAIDZ in Ubuntu server would be a nightmare for anyone that isn't an expect with the Linux CLI.

Last edited by lowfat; October 25, 2013 at 08:32 AM.
Reply With Quote
  #6 (permalink)  
Old October 25, 2013, 08:16 AM
Entilza's Avatar
Top Prospect
 
Join Date: Jun 2011
Posts: 105

My System Specs

Default

Regarding linux, I meant using samba which is pretty simple to setup. You can also install webmin for ubuntu server which gives you a web interface.
Reply With Quote
  #7 (permalink)  
Old October 25, 2013, 08:49 AM
10e's Avatar
10e 10e is offline
Allstar
F@H
 
Join Date: Jan 2008
Location: Markham, Ontario
Posts: 538

My System Specs

Default

Quote:
Originally Posted by Entilza View Post
Enjoy your esxi experiments.

Just FYI, there are many basic intel processors that offer Vt-D passthrough not just server processors.

For example the i5-3570 offers it but the i5-3570K does not. Similarly with the 4670 and 4670k etc. Intel.com can show you which processor supports. Even my 5 year old E8400 has it.

Also you can use any basic machine for esxi testing if anyone is considering it.

Finally, you can also use your ubuntu server as a file server instead of freenas.

Keep posting on what yer up to!
I'd stick to SB-E or IB-E processors with x79 or C606 chipsets, or AMD FX with 990FX, or Opteron/Xeon with server-based chipsets for this due to the PCI-E lane limitations as lowfat has said.
__________________

Xeven: How about 10^8.450980400142567e-001 -as a possible replacement for "10e"

http://www.heatware.com/eval.php?id=71732
Reply With Quote
  #8 (permalink)  
Old October 25, 2013, 09:02 AM
lowfat's Avatar
Moderator
 
Join Date: Feb 2007
Location: Grande Prairie, AB
Posts: 7,955

My System Specs

Default

Been having issues w/ write performance to the FreeNAS share. Haven't been able to break 60MB/s. Have tried a bunch of different settings but nothing has resulted in higher write performance.
Reply With Quote
  #9 (permalink)  
Old October 25, 2013, 09:27 AM
Shadowmeph's Avatar
Hall Of Fame
F@H
 
Join Date: Oct 2007
Posts: 3,353

My System Specs

Default

I was trying to use hyperviser ( server 2012 ) with my intel build and although after a bunch of problems I did get hyperviser to run but I could not install any OS's becasue apparently the i5 2500k dose not support this it supports installing Virtual box but I cannot install windows 8 or the like onto it which is why I put together my AMD 1100t be system only six core but I do know it will work. I posted else where becasue m,y AMD system was having Bios problems but I think i solved ( with the help of others on the forums here) the problem.
I do want to do this using one of my other AMD server chips but for now at least I don't have the motherboard that plays nice with the es 6234 chip.
I am going to follow this thread
Reply With Quote
  #10 (permalink)  
Old October 25, 2013, 09:33 AM
odis172's Avatar
Allstar
F@H
 
Join Date: Oct 2008
Location: Ottawa, ON
Posts: 683

My System Specs

Default

ESXI works well when installed on a usb drive or flash card, that would free up your ssd.
I'm not sure if you are planning on putting more vms on your local ssd, however a neat way to have it setup is to have esxi on a usb drive, then a small local drive for a freenas vm. You boot esxi, freenas starts, then it shares it's raid array to esxi - and the rest of your vm's start.
I played around with freenas a year ago and wasn't able to get the write speeds over ~75 MB/s - but I didn;t have a ssd caching drive. You may have more luck, but what I can tell you is that there is some comprehensive guides on storage configuration for zfs. It can get quite complicated.
This is a good read though, and the storage section has some great links
Hardware Recommendations - Freenas
__________________
Laptop: Lenovo T420 - Core i5 2520 - 8GB DDR3
Server: 2 xeon E5530 - Asus Z8N-D6 - 12GB DDR3 - 8x 500GB raid 50 - Perc 6/i - Compucase RA448A00
Reply With Quote
Reply


Thread Tools
Display Modes

Similar Threads
Thread Thread Starter Forum Replies Last Post
NAS or Home server? Mena New Builds 30 January 29, 2013 09:13 PM
home server peekeesh New Builds 8 December 26, 2012 11:32 AM
WIP Home File Server - Modding Rack mount Server Chassis CoolHandL Case Mods, Worklog & Gallery 24 December 22, 2010 11:37 AM
Home Server trainguy New Builds 8 October 5, 2008 05:16 PM
Home Server Phobia O/S's, Drivers & General Software 19 September 28, 2008 08:22 AM