s00505.nersc.gov.26418.0
powered by IPM
command: /usr/common/homes/j/bob/a2.out/a2.out.x 40000 16 16 128 4 4
codename: unknown
state: completed
username: bob
group: mpccc
host: s01813 (006035774C00_AIX)
mpi_tasks: 1024 on 64 hosts
start: 02/11/05/12:39:41
wallclock: 2.94390e+03 sec
stop: 02/11/05/13:28:19
%comm: 15.0063602451786
max memory: 748.300656 gbytes
total gflop/sec: 1416.03622
switch(send): 1969.94654275477 gbytes
switch(recv): 1963.48279621452 gbytes

Regions

Label Ntasks <MPI sec> <Wall sec> %Wall [gflop/sec]
W 1024 104.7385 2123.3360 72.13 9.4380e-01
dSdC 1024 0.0000 31.0262 1.05 0.0000e+00
invD 1024 121.8305 187.0132 6.35 3.3647e-01
io_W 1024 21.0591 30.3011 1.03 0.0000e+00
io_dSdC 1024 32.4769 46.4694 1.58 0.0000e+00
io_in_sync_W 1024 0.0025 0.0025 0.00 3.0000e-05
io_in_sync_dSdC 1024 8.0194 8.0211 0.27 0.0000e+00
io_in_sync_invD 1024 0.3231 0.3251 0.01 0.0000e+00
io_invD 1024 22.3856 32.5097 1.10 0.0000e+00
io_out_sync_W 1024 17.5369 17.5369 0.60 0.0000e+00
io_out_sync_dSdC 1024 34.8241 34.8252 1.18 0.0000e+00
io_out_sync_invD 1024 16.1677 16.1689 0.55 0.0000e+00
remap 1024 58.7248 368.0334 12.50 0.0000e+00

Computation

Event Count Pop
PM_CYC 1.6686766350473e+15 *
PM_FPU0_CMPL 1.0450057398122e+15 *
PM_FPU1_CMPL 1.04348046159518e+15 *
PM_FPU_FMA 2.08018291415355e+15 *
PM_INST_CMPL 3.66755569560868e+15 *
PM_LD_CMPL 1.16684473955345e+15 *
PM_ST_CMPL 849924220757493 *
PM_TLB_MISS 484913554455 *

Communication

% of MPI Time
Compute Event Statistics
  Ntasks Avg Min Max
PM_CYC * 1629567026413.38 1613248978120 (904) 1657403988716 (240)
PM_FPU0_CMPL * 1020513417785.35 1013236356821 (857) 1043390741049 (0)
PM_FPU1_CMPL * 1019023888276.54 1011763974973 (857) 1041921735931 (0)
PM_FPU_FMA * 2031428627103.08 2016980030224 (857) 2077006100299 (0)
PM_INST_CMPL * 3581597358992.85 3549072413864 (189) 3648932586948 (0)
PM_LD_CMPL * 1139496815970.16 1128792625873 (187) 1160574691133 (0)
PM_ST_CMPL * 830004121833.49 820474011530 (475) 845569837198 (240)
PM_TLB_MISS * 473548393.02 423469796 (231) 1202821925 (432)
Communication Event Statistics (100.00% detail, -2.9676e-02 error)
  Buffer Size Ncalls Total Time Min Time Max Time %MPI %Wall
MPI_Recv 4 359648 92703.293 0.000 42.036 20.49 3.08
MPI_Barrier 0 71680 78942.319 0.000 36.085 17.45 2.62
MPI_Recv 1277952 2082528 49262.391 0.002 17.303 10.89 1.63
MPI_Send 1277952 1960000 42308.944 0.003 33.675 9.35 1.40
MPI_Bcast 199360512 784 41575.117 19.092 67.781 9.19 1.38
MPI_Bcast 201916416 224 13089.703 42.333 67.794 2.89 0.43
MPI_Reduce 294912 247424 7932.026 0.003 0.251 1.75 0.26
MPI_Reduce 262144 258048 7390.395 0.003 0.292 1.63 0.25
MPI_Recv 1294336 297472 6957.175 0.002 22.643 1.54 0.23
MPI_Reduce 196608 258048 6823.380 0.002 1.457 1.51 0.23
MPI_Reduce 229376 258048 6362.571 0.002 0.238 1.41 0.21
MPI_Send 1294336 280000 5804.218 0.003 23.001 1.28 0.19
MPI_Recv 1179648 59768 5525.938 0.003 1.833 1.22 0.18
MPI_Reduce 131072 258048 5075.142 0.001 0.820 1.12 0.17
MPI_Reduce 163840 258048 4781.784 0.002 0.194 1.06 0.16
MPI_Recv 1048576 61504 4740.601 0.003 1.689 1.05 0.16
MPI_Reduce 98304 258048 4348.428 0.001 0.878 0.96 0.14
MPI_Recv 917504 61504 4056.346 0.003 1.341 0.90 0.13
MPI_Reduce 48 9216 3536.725 0.000 11.891 0.78 0.12
MPI_Recv 786432 61504 3394.443 0.002 1.045 0.75 0.11
MPI_Reduce 65536 258048 2830.357 0.000 0.168 0.63 0.09
MPI_Recv 655360 61527 2784.269 0.002 0.716 0.62 0.09
MPI_Reduce 8 1024 2499.763 0.000 15.044 0.55 0.08
MPI_Reduce 327680 64896 2443.164 0.003 0.271 0.54 0.08
MPI_Bcast 294912 123392 2171.329 0.001 0.125 0.48 0.07
MPI_Recv 524288 61504 2083.081 0.002 0.545 0.46 0.07
MPI_Bcast 262144 126976 2038.065 0.001 0.175 0.45 0.07
MPI_Recv 1310720 17112 2020.860 0.004 0.906 0.45 0.07
MPI_Recv 11796480 5675 1769.316 0.027 16.893 0.39 0.06
MPI_Bcast 229376 126976 1734.626 0.001 0.088 0.38 0.06
MPI_Reduce 32768 267872 1595.331 0.000 1.010 0.35 0.05
MPI_Recv 393216 61504 1555.912 0.001 0.268 0.34 0.05
MPI_Bcast 196608 126976 1540.856 0.001 1.451 0.34 0.05
MPI_Recv 262144 188480 1525.966 0.001 0.309 0.34 0.05
MPI_Recv 131072 189442 1472.337 0.000 0.389 0.33 0.05
MPI_Recv 13107200 9712 1330.880 0.030 13.344 0.29 0.04
MPI_Bcast 163840 126976 1307.713 0.001 0.064 0.29 0.04
MPI_Recv 196608 128960 1298.764 0.001 1.447 0.29 0.04
MPI_Bcast 131072 126976 1101.723 0.000 0.047 0.24 0.04
MPI_Recv 229376 126976 1027.656 0.001 1.255 0.23 0.03
MPI_Bcast 32768 126848 978.648 0.000 0.892 0.22 0.03
MPI_Bcast 204505088 16 969.601 47.575 66.079 0.21 0.03
MPI_Recv 163840 126976 954.999 0.000 1.112 0.21 0.03
MPI_Bcast 98304 126976 878.957 0.000 0.050 0.19 0.03
MPI_Send 294912 117428 822.955 0.001 0.628 0.18 0.03
MPI_Send 196608 124000 817.949 0.001 0.661 0.18 0.03
MPI_Bcast 65536 126976 787.680 0.000 0.041 0.17 0.03
MPI_Send 229376 123008 738.306 0.001 1.101 0.16 0.02
MPI_Recv 98304 126976 721.983 0.000 0.820 0.16 0.02
MPI_Bcast 327680 35328 671.833 0.001 0.128 0.15 0.02
MPI_Recv 65536 128931 624.587 0.000 0.547 0.14 0.02
MPI_Recv 32768 155101 613.313 0.000 0.524 0.14 0.02
MPI_Send 262144 153760 611.993 0.001 0.564 0.14 0.02
MPI_Recv 294912 129878 606.437 0.001 0.534 0.13 0.02
MPI_Send 98304 123008 586.371 0.000 0.566 0.13 0.02
MPI_Recv 10616832 826 584.513 0.039 6.348 0.13 0.02
MPI_Send 131072 154721 580.723 0.000 0.620 0.13 0.02
MPI_Recv 327680 60992 533.927 0.001 0.553 0.12 0.02
MPI_Send 163840 123008 499.481 0.001 0.536 0.11 0.02
MPI_Bcast 4 10016 469.902 0.000 0.438 0.10 0.02
MPI_Send 32768 151125 450.911 0.000 1.028 0.10 0.01
MPI_Send 65536 124000 450.174 0.000 0.398 0.10 0.01
MPI_Send 327680 29636 317.893 0.001 0.722 0.07 0.01
MPI_Recv 11206656 237 303.042 0.045 5.880 0.07 0.01
 
Load balance by task: HPM counters
by MPI rank, by MPI time
Load balance by task: memory, flops, timings
by MPI rank, by MPI time
 
Communication balance by task (sorted by MPI rank)
Communication balance by task (sorted by MPI time)
 
Communication Topology : point to point data flow
time spent , ~rate , map_data file map_adjacency file
Communication Topology : Connectivity (sorted by # neighbors)
 
Message Buffer Size Distributions: time
abs, hist
Message Buffer Size Distributions: Ncalls
abs, hist
Message Buffer Size Distributions: data volume
abs, hist
Switch volume by node
 
Memory usage by node


powered by IPM