Wednesday, 12 October 2011

EqualLogic PS6500E speed test with linux hosts

Equipment
2 linux hosts, each comprising:
  • Dell R610, 96GB RAM, 2x Intel Xeon Processor X5680 (12M Cache, 3.33 GHz, 6.40 GT/s) (6 cores, 12 hyperthreads), 2 x 4 port Intel Pro1000 NICs  (Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01))
  • 2 x Dell PC6224 stacked gigabit switches
  • 1 x EQL PS6500E with 48 x 1TB SATA 7200RPM drives ST31000524NS SATA 3Gb/s 1TB 32MB cache
  • Wiring: 4 x gig from each host to PC6224s split across switches; 8 x gig from both PS6500E controllers to swiches, split across switches.
  • Switches and hosts confirgured for MTU of 9000 to match PS6500
Optimise both hosts thus:
echo 4194304 > /proc/sys/net/core/rmem_default
echo 4194304 > /proc/sys/net/core/rmem_max
echo 4194304 > /proc/sys/net/core/wmem_default
echo 4194304 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling
echo "4194304 4194304 4194304" > /proc/sys/net/ipv4/tcp_rmem
echo "4194304 4194304 4194304" > /proc/sys/net/ipv4/tcp_wmem
ifconfig eth8 txqueuelen 1000
ifconfig eth9 txqueuelen 1000
ifconfig eth10 txqueuelen 1000
ifconfig eth11 txqueuelen 1000

PS6500E has 2 x 10TB volumes ona RAID50, each volume is mounted to one of the hosts using round robin multipathing, formatted to XFS and mounted:

iscsiadm -m node --logoutall=all
iscsiadm -m discovery -t st -p 10.223.10.5
iscsiadm -m node --login -T 'iqn.2001-05.com.equallogic:0-8a0906-388f3f80a-e907d15ce794e948-test3'
multipath
mkfs.xfs -L TEST3 /dev/dm-0
mount -onoatime,logbsize=262144,logbufs=8 /dev/dm-0 /mnt/

We used streamput and streamget which are simple homegrown C programs to read and write single large buffers of random data to files (few large files), run for a reasonable time to ensure the host RAM is saturated vis a vis caches (streamput can also do O_DIRECT writes to bypass caching). Buffer for single IO operation is 1MB of random data, generated once at program start. Buffer is page aligned for optimal DMA and O_DIRECT.

Our basic streamput test comprised:
./streamput -v -t 1200 -l 4096 -w /mnt/<uniquesubdir>

(Write 4GB files with random name into test dir and repeat for a total of 1200 seconds using plain ordinary C file writes with no special flags)

Our basic streamread test comprised
./streamread -v -t 1200 -w /mnt/<uniquesubdir>

(Read 4GB files with random name from test dir and repeat for a total of 1200 seconds using plain ordinary C file writes with no special flags)


Test 1 simple streamed writing
2 x streamput tests on 2 hosts in parallel

Results

streamput,PID,14225,Megabytes/sec,45.8
streamput,PID,14227,Megabytes/sec,45.0
streamput,PID,13640,Megabytes/sec,44.1
streamput,PID,13639,Megabytes/sec,44.1

Total 179MB/sec

Test 2 reading
2 x streamread tests on 2 hosts in parallel

Results
streamread,PID,14364,Megabytes/sec,54.5
streamread,PID,14363,Megabytes/sec,50.9
streamread,PID,13767,Megabytes/sec,54.6
streamread,PID,13766,Megabytes/sec,64.5

Total 224.5MB/sec

Reconfigure to RAID10, setup with XFS s above.

Test 3
2 x streamput tests on 2 hosts in parallel

Results

streamput,PID,15107,Megabytes/sec,46.1
streamput,PID,15106,Megabytes/sec,48.6
streamput,PID,14471,Megabytes/sec,44.2
streamput,PID,14470,Megabytes/sec,50.8

Total 189.7MB/sec

Test 4

echo 3 > /proc/sys/vm/drop_caches
2 x streamread tests on 2 hosts in parallel

Results

streamread,PID,15197,Megabytes/sec,19.9
streamread,PID,15198,Megabytes/sec,20.0
streamread,PID,14558,Megabytes/sec,23.4
streamread,PID,14557,Megabytes/sec,23.1

Total 86.4 MB/sec

Reconfigure to RAID10, setup with XFS s above.

Test 5
2 x streamread tests on 2 hosts in parallel

Results

streamput,PID,15845,Megabytes/sec,40.8
streamput,PID,15843,Megabytes/sec,49.7
streamput,PID,15063,Megabytes/sec,43.6
streamput,PID,15062,Megabytes/sec,45.9

Total 180MB/sec

Test 6

echo 3 > /proc/sys/vm/drop_caches
2 x streamread tests on 2 hosts in parallel


streamread,PID,15088,Megabytes/sec,72.0
streamread,PID,15087,Megabytes/sec,45.6
streamread,PID,15878,Megabytes/sec,49.4
streamread,PID,15877,Megabytes/sec,35.9

Total 202.9MB/sec


Reconfigure to RAID10, setup with XFS s above.

Test 7 during RAID5 reconstruction


echo 3 > /proc/sys/vm/drop_caches
2 x streamread tests on 2 hosts in parallel


Results

streamput,PID,15379,Megabytes/sec,43.4
streamput,PID,15378,Megabytes/sec,48.8
streamput,PID,16235,Megabytes/sec,44.7
streamput,PID,16234,Megabytes/sec,47.9

Total 184.8MB/sec

Test 8 Array reconstructing

echo 3 > /proc/sys/vm/drop_caches
2 x streamread tests on 2 hosts in parallel


Results

streamread,PID,15398,Megabytes/sec,45.6
streamread,PID,15397,Megabytes/sec,44.4
streamread,PID,16253,Megabytes/sec,36.5
streamread,PID,16252,Megabytes/sec,36.8

Total 163.3MB/sec

Array (re)construction time:

Time to initialise RAID5: 58% completion in approx 14 hours. At 13:05, 62%, 15:29 73%. Estimated full time 21 hours 49 minutes


fio IOPS tests

Test 9 RAID5 (Raid rebuild complete)

fio --filename=/dev/dm-0 --direct=1 --rw=randwrite --bs=4k --numjobs=64 --runtime=300 --group_reporting --name=raid5

on 1 host

IOPS=2862

raid5: (groupid=0, jobs=64): err= 0: pid=21151
  write: io=3407MB, bw=11452KB/s, iops=2862, runt=304630msec
    clat (usec): min=220, max=7629K, avg=21083.39, stdev=47412.87
    bw (KB/s) : min=    0, max= 5261, per=2.36%, avg=269.97, stdev=69.20
  cpu          : usr=0.04%, sys=0.10%, ctx=874983, majf=0, minf=1523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/872151, short=0/0
     lat (usec): 250=2.28%, 500=90.23%, 750=3.62%, 1000=0.47%
     lat (msec): 2=2.27%, 4=0.53%, 10=0.09%, 20=0.06%, 50=0.05%
     lat (msec): 100=0.02%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.33%


Run status group 0 (all jobs):
  WRITE: io=3407MB, aggrb=11451KB/s, minb=11726KB/s, maxb=11726KB/s, mint=304630msec, maxt=304630msec


Disk stats (read/write):
  dm-0: ios=0/871635, merge=0/0, ticks=0/19261868, in_queue=19261440, util=99.97%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%

Test 10 RAID5 write (Raid rebuild complete) shorter test time

fio --filename=/dev/dm-0 --direct=1 --rw=randwrite --bs=4k --numjobs=64 --runtime=60 --group_reporting --name=raid5

IOPS=3261

raid5: (groupid=0, jobs=64): err= 0: pid=21222
  write: io=839936KB, bw=13048KB/s, iops=3261, runt= 64375msec
    clat (usec): min=222, max=7480K, avg=18589.17, stdev=43118.68
    bw (KB/s) : min=    0, max= 7520, per=1.55%, avg=202.70, stdev=13.74
  cpu          : usr=0.11%, sys=0.10%, ctx=211492, majf=0, minf=1523
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/209984, short=0/0
     lat (usec): 250=11.03%, 500=83.24%, 750=2.35%, 1000=0.52%
     lat (msec): 2=1.74%, 4=0.54%, 10=0.09%, 20=0.06%, 50=0.05%
     lat (msec): 100=0.02%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.03%, >=2000=0.29%


Run status group 0 (all jobs):
  WRITE: io=839936KB, aggrb=13047KB/s, minb=13360KB/s, maxb=13360KB/s, mint=64375msec, maxt=64375msec


Disk stats (read/write):
  dm-0: ios=0/209453, merge=0/0, ticks=0/3930736, in_queue=3930732, util=99.61%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%


Test 10 RAID5 read

fio --filename=/dev/dm-0 --direct=1 --rw=randread --bs=4k --numjobs=64 --runtime=60 --group_reporting --name=raid5


IOPS=159

raid5: (groupid=0, jobs=64): err= 0: pid=21295
  read : io=40636KB, bw=654080B/s, iops=159, runt= 63618msec
    clat (usec): min=527, max=6767K, avg=374501.76, stdev=152589.35
    bw (KB/s) : min=    0, max=  318, per=2.21%, avg=14.08, stdev= 1.64
  cpu          : usr=0.09%, sys=0.01%, ctx=10210, majf=0, minf=1651
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=10159/0, short=0/0
     lat (usec): 750=0.11%, 1000=0.15%
     lat (msec): 2=0.97%, 4=9.81%, 10=55.35%, 20=17.33%, 50=1.62%
     lat (msec): 100=0.83%, 250=2.15%, 500=1.53%, 750=0.88%, 1000=0.64%
     lat (msec): 2000=1.39%, >=2000=7.24%


Run status group 0 (all jobs):
   READ: io=40636KB, aggrb=638KB/s, minb=654KB/s, maxb=654KB/s, mint=63618msec, maxt=63618msec


Disk stats (read/write):
  dm-0: ios=10138/0, merge=0/0, ticks=3905828/0, in_queue=3905812, util=99.62%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%

Test 10 RAID5 read on 2 hosts in parallel

fio --filename=/dev/dm-0 --direct=1 --rw=randread --bs=4k --numjobs=64 --runtime=60 --group_reporting --name=raid5

IOPS=138 and 123, total 261

Test 11 RAID 10 Write (Reconstructing) 1 Host
fio --filename=/dev/dm-0 --direct=1 --rw=randwrite --bs=4k --numjobs=64 --runtime=60 --group_reporting --name=raid10

IOPS=3068

Test 12 RAID 10 Read (Reconstructing) 1 Host
fio --filename=/dev/dm-0 --direct=1 --rw=randread --bs=4k --numjobs=64 --runtime=60 --group_reporting --name=raid10

IOPS=8
(Seems wrong but repeatible!)

1 comment:

  1. Any chance you could publish your streamput/streamread code? I'd be interested in testing against our SAS PS6000.
    Thanks

    ReplyDelete