Thanks for the heads up and vote of confidence guys!
I have configured VSAN on the above mentioned hardware and am pleasantly surprised by its genius. It was not as hard as I expected it to be considering hundreds of pages of documentation and hardware requirements to sift through.
I have a few more questions if you do not mind answering them...
1) I have not exactly allocated the recommended 1:10 flash to magnetic storage ratio - how can I expect this to impact performance in future? Especially if I go adding another 6 x 4TB magnetic drives with my 800GB SSD?
2) What is the most effective way for me to benchtest and get performance metrics regarding the VSANs performance under stress?
3) Currently the OEM whitebox runs the VCSA and the VSAN Witness Appliance. Is this a normal configuration? where do people normally put the VCSA in a 3 node cluster like this?
4) the workloads this will be doing will be heavily pushed by consistent and unrelenting CCTV footage streams in (and sometimes out for playback)... what would be your recommendations for avoiding issues such as SATA queue depth, or any other bottlenecks you can see? The DL380's have 4x1Gbps NICs + a 2 Port 10GbE card in them for future use when this system expands. LACP may be employed if needed depending on budget availability for 10GbE Switches.
5) Can someone please explain to me or point me to a guide where the VSAN Witness components are explained in lay-mans terms? I fail to understand or see where/why the witness components come into play in the greater scheme of things and if the performance of the Witness affects the performance of the VSAN?
6) any suggestions for not breaking things when I test the HA?
More thanks in advance!