128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)
128m: Laying out IO file(s) (1 file(s) / 128MB)

128m: (groupid=0, jobs=32): err= 0: pid=534
  read : io=1697.8MB, bw=173794KB/s, iops=43448 , runt= 10003msec
    clat (usec): min=28 , max=352810 , avg=732.49, stdev=3037.59
     lat (usec): min=28 , max=352810 , avg=733.15, stdev=3037.60
    clat percentiles (usec):
     |  1.00th=[  350],  5.00th=[  462], 10.00th=[  516], 20.00th=[  572],
     | 30.00th=[  604], 40.00th=[  636], 50.00th=[  676], 60.00th=[  724],
     | 70.00th=[  764], 80.00th=[  820], 90.00th=[  900], 95.00th=[ 1004],
     | 99.00th=[ 1336], 99.50th=[ 1560], 99.90th=[ 2576], 99.95th=[ 3504],
     | 99.99th=[28288]
    bw (KB/s)  : min= 1815, max= 5880, per=3.13%, avg=5435.91, stdev=826.27
    lat (usec) : 50=0.13%, 100=0.04%, 250=0.09%, 500=7.63%, 750=58.41%
    lat (usec) : 1000=28.57%
    lat (msec) : 2=4.93%, 4=0.16%, 10=0.02%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%, 500=0.01%
  cpu          : usr=0.49%, sys=2.37%, ctx=435498, majf=0, minf=1017
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=434615/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1697.8MB, aggrb=173793KB/s, minb=173793KB/s, maxb=173793KB/s, mint=10003msec, maxt=10003msec

Disk stats (read/write):
  xvda: ios=434255/5, merge=0/0, ticks=31663/14, in_queue=31399, util=87.79%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=569
  read : io=1724.8MB, bw=176494KB/s, iops=44123 , runt= 10007msec
    clat (usec): min=22 , max=98250 , avg=719.79, stdev=882.21
     lat (usec): min=22 , max=98251 , avg=720.73, stdev=882.23
    clat percentiles (usec):
     |  1.00th=[  294],  5.00th=[  454], 10.00th=[  524], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  644], 50.00th=[  676], 60.00th=[  724],
     | 70.00th=[  772], 80.00th=[  828], 90.00th=[  916], 95.00th=[ 1020],
     | 99.00th=[ 1464], 99.50th=[ 1880], 99.90th=[ 2864], 99.95th=[ 3920],
     | 99.99th=[22144]
    bw (KB/s)  : min= 4345, max= 5956, per=3.11%, avg=5496.13, stdev=280.84
    lat (usec) : 50=0.57%, 100=0.06%, 250=0.15%, 500=7.15%, 750=58.14%
    lat (usec) : 1000=28.40%
    lat (msec) : 2=5.12%, 4=0.36%, 10=0.03%, 20=0.01%, 50=0.01%
    lat (msec) : 100=0.01%
  cpu          : usr=0.57%, sys=2.37%, ctx=441942, majf=0, minf=1047
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=441543/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1724.8MB, aggrb=176493KB/s, minb=176493KB/s, maxb=176493KB/s, mint=10007msec, maxt=10007msec

Disk stats (read/write):
  xvda: ios=434657/4, merge=0/0, ticks=26771/21, in_queue=26581, util=76.27%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=605
  read : io=1743.3MB, bw=178437KB/s, iops=44609 , runt= 10004msec
    clat (usec): min=19 , max=23499 , avg=711.50, stdev=302.71
     lat (usec): min=20 , max=23500 , avg=712.49, stdev=302.80
    clat percentiles (usec):
     |  1.00th=[  278],  5.00th=[  462], 10.00th=[  532], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  636], 50.00th=[  668], 60.00th=[  716],
     | 70.00th=[  764], 80.00th=[  828], 90.00th=[  908], 95.00th=[ 1012],
     | 99.00th=[ 1496], 99.50th=[ 1880], 99.90th=[ 2832], 99.95th=[ 4256],
     | 99.99th=[12352]
    bw (KB/s)  : min= 5232, max= 5792, per=3.11%, avg=5556.31, stdev=99.29
    lat (usec) : 20=0.01%, 50=0.65%, 100=0.06%, 250=0.16%, 500=6.46%
    lat (usec) : 750=59.89%, 1000=27.46%
    lat (msec) : 2=4.91%, 4=0.36%, 10=0.04%, 20=0.01%, 50=0.01%
  cpu          : usr=0.68%, sys=2.28%, ctx=446439, majf=0, minf=1073
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=446270/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1743.3MB, aggrb=178436KB/s, minb=178436KB/s, maxb=178436KB/s, mint=10004msec, maxt=10004msec

Disk stats (read/write):
  xvda: ios=439109/4, merge=0/0, ticks=23631/18, in_queue=23481, util=68.74%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=640
  read : io=1746.2MB, bw=178756KB/s, iops=44688 , runt= 10003msec
    clat (usec): min=21 , max=21637 , avg=710.40, stdev=304.27
     lat (usec): min=21 , max=21637 , avg=711.46, stdev=304.30
    clat percentiles (usec):
     |  1.00th=[  278],  5.00th=[  462], 10.00th=[  532], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  644], 50.00th=[  676], 60.00th=[  716],
     | 70.00th=[  764], 80.00th=[  828], 90.00th=[  908], 95.00th=[ 1004],
     | 99.00th=[ 1480], 99.50th=[ 1864], 99.90th=[ 2800], 99.95th=[ 3856],
     | 99.99th=[13376]
    bw (KB/s)  : min= 5176, max= 5864, per=3.11%, avg=5562.63, stdev=123.43
    lat (usec) : 50=0.67%, 100=0.04%, 250=0.17%, 500=6.49%, 750=59.98%
    lat (usec) : 1000=27.45%
    lat (msec) : 2=4.79%, 4=0.34%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.61%, sys=2.33%, ctx=447312, majf=0, minf=1041
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=447023/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1746.2MB, aggrb=178755KB/s, minb=178755KB/s, maxb=178755KB/s, mint=10003msec, maxt=10003msec

Disk stats (read/write):
  xvda: ios=439609/6, merge=0/0, ticks=23939/19, in_queue=23777, util=70.43%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=675
  read : io=1760.3MB, bw=180212KB/s, iops=45052 , runt= 10002msec
    clat (usec): min=21 , max=25481 , avg=704.68, stdev=259.58
     lat (usec): min=21 , max=25481 , avg=705.67, stdev=259.65
    clat percentiles (usec):
     |  1.00th=[  286],  5.00th=[  462], 10.00th=[  532], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  636], 50.00th=[  660], 60.00th=[  700],
     | 70.00th=[  756], 80.00th=[  820], 90.00th=[  900], 95.00th=[  996],
     | 99.00th=[ 1480], 99.50th=[ 1944], 99.90th=[ 2800], 99.95th=[ 3376],
     | 99.99th=[ 8384]
    bw (KB/s)  : min= 5336, max= 5976, per=3.12%, avg=5619.20, stdev=144.03
    lat (usec) : 50=0.62%, 100=0.05%, 250=0.17%, 500=6.36%, 750=61.83%
    lat (usec) : 1000=26.06%
    lat (msec) : 2=4.45%, 4=0.43%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.66%, sys=2.29%, ctx=450898, majf=0, minf=1041
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=450620/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1760.3MB, aggrb=180211KB/s, minb=180211KB/s, maxb=180211KB/s, mint=10002msec, maxt=10002msec

Disk stats (read/write):
  xvda: ios=443804/1, merge=0/0, ticks=23863/18, in_queue=23701, util=68.96%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=715
  read : io=1753.8MB, bw=179508KB/s, iops=44877 , runt= 10004msec
    clat (usec): min=25 , max=33498 , avg=707.53, stdev=264.71
     lat (usec): min=25 , max=33499 , avg=708.56, stdev=264.75
    clat percentiles (usec):
     |  1.00th=[  298],  5.00th=[  458], 10.00th=[  524], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  636], 50.00th=[  668], 60.00th=[  708],
     | 70.00th=[  756], 80.00th=[  820], 90.00th=[  908], 95.00th=[ 1004],
     | 99.00th=[ 1496], 99.50th=[ 1976], 99.90th=[ 2896], 99.95th=[ 3440],
     | 99.99th=[ 7840]
    bw (KB/s)  : min= 5232, max= 5896, per=3.12%, avg=5599.17, stdev=118.03
    lat (usec) : 50=0.49%, 100=0.08%, 250=0.20%, 500=6.95%, 750=60.39%
    lat (usec) : 1000=26.69%
    lat (msec) : 2=4.72%, 4=0.45%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.62%, sys=2.33%, ctx=449154, majf=0, minf=1041
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=448950/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1753.8MB, aggrb=179508KB/s, minb=179508KB/s, maxb=179508KB/s, mint=10004msec, maxt=10004msec

Disk stats (read/write):
  xvda: ios=442313/4, merge=0/0, ticks=24800/16, in_queue=24633, util=71.95%
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
128m: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 2.0.7
Starting 32 processes

128m: (groupid=0, jobs=32): err= 0: pid=755
  read : io=1766.7MB, bw=180853KB/s, iops=45213 , runt= 10003msec
    clat (usec): min=16 , max=28274 , avg=702.02, stdev=337.19
     lat (usec): min=17 , max=28275 , avg=702.99, stdev=337.28
    clat percentiles (usec):
     |  1.00th=[  282],  5.00th=[  458], 10.00th=[  524], 20.00th=[  580],
     | 30.00th=[  612], 40.00th=[  628], 50.00th=[  660], 60.00th=[  692],
     | 70.00th=[  748], 80.00th=[  820], 90.00th=[  900], 95.00th=[  996],
     | 99.00th=[ 1480], 99.50th=[ 1864], 99.90th=[ 2864], 99.95th=[ 3696],
     | 99.99th=[10688]
    bw (KB/s)  : min= 5136, max= 5896, per=3.11%, avg=5632.63, stdev=117.73
    lat (usec) : 20=0.01%, 50=0.62%, 100=0.06%, 250=0.17%, 500=6.76%
    lat (usec) : 750=62.62%, 1000=24.84%
    lat (msec) : 2=4.52%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=0.66%, sys=2.29%, ctx=452482, majf=0, minf=1044
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=452268/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=1766.7MB, aggrb=180852KB/s, minb=180852KB/s, maxb=180852KB/s, mint=10003msec, maxt=10003msec

Disk stats (read/write):
  xvda: ios=445131/4, merge=0/0, ticks=24812/25, in_queue=24667, util=69.08%
