Errors with big arrays and many communicators #556
Description
I ran into some errors using the julia_set benchmark:
First, I tried to shoot the moon and see if we could go up to 16384:
$ mpiexec -n 1 python benchmark_julia.py -r6 -olong_run.json 128 256 512 1024 204
8 4096 8192 16384 : -n 12 python -m distarray.apps.engine
...
(343/2352: 0.895s) (1406335077.099205, 1406335077.99378, 'c-c', 8192, '(-0.045+0.45j)', 12, [174749288L, 174751472L, 174748776L, 174752138L, 174748831L, 174749623L, 174751680L,
174749623L, 174747481L, 174750332L, 174746970L, 174749666L])
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 118, in func_call
self._INTERCOMM.send(res, dest=self.client_rank)
File "Comm.pyx", line 800, in mpi4py.MPI.Comm.send (src/mpi4py.MPI.c:66431)
File "pickled.pxi", line 180, in mpi4py.MPI.PyMPI_send (src/mpi4py.MPI.c:27319)
File "pickled.pxi", line 81, in mpi4py.MPI._p_Pickle.dump (src/mpi4py.MPI.c:26188)
SystemError: NULL result without error in PyObject_Call
Notice that it handled all the runs up through 8192x8192
Then, I tried to just do a full run of 6 reps of up to 8192x8192:
$ mpiexec -n 1 python benchmark_julia.py -r6 -olong_run_2.json 128 256 512 1024 2
048 4096 8192 : -n 12 python -m distarray.apps.engine
...
(2042/2058: 1.267s) (1406336596.783966, 1406336598.050912, 'c-c', 8192, '(-0.045+0.45j)', 8, [262124206L, 262123056L, 262123488L, 262123056L, 262121394L, 262128371L, 262123938L, 262128371L])
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/rgrant/development/distarray/distarray/apps/engine.py", line 9, in <module>
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
distribution = Distribution(comm=comm, dim_data=dim_data)
Engine()
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 41, in __init__
val = self.parse_msg(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 80, in parse_msg
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40c00cd, ndims=2, dims=0x472c2b0, periods=0x472b330, reorder=0, comm_cart=0x7f8c54715f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
ret = func(msg)
File "/home/rgrant/development/distarray/distarray/mpi_engine.py", line 115, in func_call
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40800f5, ndims=2, dims=0x16daf80, periods=0x16e0860, reorder=0, comm_cart=0x7f6f0202ff70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc4060065, ndims=2, dims=0x118b0a0, periods=0x118e0f0, reorder=0, comm_cart=0x7f5486209f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc405001d, ndims=2, dims=0x1d38040, periods=0x1d438b0, reorder=0, comm_cart=0x7fd9f8be9f58) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception: reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
res = new_func(*args, **kwargs)
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 133, in create_local
Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc41500ef, ndims=2, dims=0x29bdb70, periods=0x29b7820, reorder=0, comm_cart=0x7fa0ac697f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exceptionmpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40700ad, ndims=2, dims=0x2ece820, periods=0x2ece840, reorder=0, comm_cart=0x7f718ce77f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40e0015, ndims=2, dims=0x1a163f0, periods=0x1a16410, reorder=0, comm_cart=0x7ffa9190af70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40b0085, ndims=2, dims=0x2cec520, periods=0x2ceb490, reorder=0, comm_cart=0x7f431c19bf70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
distribution = Distribution(comm=comm, dim_data=dim_data)
File "/home/rgrant/development/distarray/distarray/local/maps.py", line 51, in __init__
self.comm = construct.init_comm(self.base_comm, self.grid_shape)
File "/home/rgrant/development/distarray/distarray/local/construct.py", line 38, in init_comm
reorder=False)
File "Comm.pyx", line 952, in mpi4py.MPI.Intracomm.Create_cart (src/mpi4py.MPI.c:69241)
mpi4py.MPI.Exception: Other MPI error, error stack:
PMPI_Cart_create(313).........: MPI_Cart_create(comm=0xc40a003d, ndims=2, dims=0x11f30b0, periods=0x11f84f0, reorder=0, comm_cart=0x7fbf2c609f70) failed
MPIR_Cart_create_impl(189)....:
MPIR_Cart_create(115).........:
MPIR_Comm_copy(967)...........:
MPIR_Get_contextid(521).......:
MPIR_Get_contextid_sparse(752): Too many communicators
It got much further, but gave a different error.
Then, I tried out a stress test with the numpy fancy-indexing kernel using an IPythonContext:
$ python benchmark_julia.py -o11000_ipython.json -r6 --kernel=fancy 11000
(n/n_runs: time) ('Start', 'End', 'Dist', 'Resolution', 'c', 'Engines', 'Iters')
(1/294: 349.567s) (1406435193.460827, 1406435543.028039, 'numpy', 11000, '(-0.045+0.45j)', 1, [3696797358])
(2/294: 345.118s) (1406435543.735211, 1406435888.852837, 'b-n', 11000, '(-0.045+0.45j)', 1, [3696797358])
(3/294: 347.838s) (1406435889.605796, 1406436237.443562, 'c-n', 11000, '(-0.045+0.45j)', 1, [3696797358])
(4/294: 359.565s) (1406436238.182001, 1406436597.746668, 'b-b', 11000, '(-0.045+0.45j)', 1, [3696797358])
(5/294: 354.985s) (1406436598.521989, 1406436953.507226, 'c-c', 11000, '(-0.045+0.45j)', 1, [3696797358])
(6/294: 180.324s) (1406436954.102975, 1406437134.426741, 'b-n', 11000, '(-0.045+0.45j)', 2, [1848009514, 1848787844])
(7/294: 183.050s) (1406437134.911213, 1406437317.960948, 'c-n', 11000, '(-0.045+0.45j)', 2, [1848392664, 1848404694])
(8/294: 184.815s) (1406437318.429465, 1406437503.244464, 'b-b', 11000, '(-0.045+0.45j)', 2, [1848100010, 1848697348])
(9/294: 181.443s) (1406437503.726369, 1406437685.168943, 'c-c', 11000, '(-0.045+0.45j)', 2, [1848396939, 1848400419])
(10/294: 158.055s) (1406437685.604622, 1406437843.659871, 'b-n', 11000, '(-0.045+0.45j)', 3, [685995820, 2324806074, 685995464])
(11/294: 124.062s) (1406437844.07946, 1406437968.141083, 'c-n', 11000, '(-0.045+0.45j)', 3, [1232260508, 1232276269, 1232260581])
(12/294: 141.303s) (1406437968.539321, 1406438109.841954, 'b-b', 11000, '(-0.045+0.45j)', 3, [748452256, 2199892627, 748452475])
(13/294: 128.234s) (1406438110.242824, 1406438238.477027, 'c-c', 11000, '(-0.045+0.45j)', 3, [1232262847, 1232271595, 1232262916])
(14/294: 107.972s) (1406438238.862006, 1406438346.834492, 'b-n', 11000, '(-0.045+0.45j)', 4, [289093817, 1558915697, 1559344553, 289443291])
(15/294: 97.990s) (1406438347.255682, 1406438445.245199, 'c-n', 11000, '(-0.045+0.45j)', 4, [924200092, 924202310, 924192572, 924202384])
(16/294: 101.836s) (1406438445.615707, 1406438547.451687, 'b-b', 11000, '(-0.045+0.45j)', 4, [655791340, 1192218174, 1192308670, 656479174])
(17/294: 103.377s) (1406438547.827442, 1406438651.204403, 'c-c', 11000, '(-0.045+0.45j)', 4, [924201433, 924191231, 924195506, 924209188])
(18/294: 91.840s) (1406438652.991555, 1406438744.831066, 'b-n', 11000, '(-0.045+0.45j)', 5, [127608174, 953151016, 1534681258, 953539801, 127817109])
(19/294: 88.682s) (1406438745.43266, 1406438834.114253, 'c-n', 11000, '(-0.045+0.45j)', 5, [739371975, 739361284, 739351486, 739351267, 739361346])
(20/294: 91.953s) (1406438834.955011, 1406438926.907979, 'b-b', 11000, '(-0.045+0.45j)', 5, [60876925, 1158242038, 1257966037, 1158620346, 61092012])
(21/294: 94.120s) (1406438927.43213, 1406439021.5518, 'c-c', 11000, '(-0.045+0.45j)', 5, [739364921, 739355182, 739361005, 739360920, 739355330])
(22/294: 83.065s) (1406439022.584114, 1406439105.6491, 'b-n', 11000, '(-0.045+0.45j)', 6, [63135170, 623389934, 1163042463, 1162292639, 622223585, 62713567])
(23/294: 72.385s) (1406439106.217274, 1406439178.601975, 'c-n', 11000, '(-0.045+0.45j)', 6, [616129929, 616143599, 616130065, 616130579, 616132670, 616130516])
(24/294: 84.711s) (1406439179.159553, 1406439263.870242, 'b-b', 11000, '(-0.045+0.45j)', 6, [213752450, 1099763033, 534494031, 534699806, 1100129594, 213958444])
(25/294: 78.355s) (1406439264.387081, 1406439342.742386, 'c-c', 11000, '(-0.045+0.45j)', 6, [616130702, 616131119, 616130843, 616132145, 616140476, 616132073])
(26/294: 75.927s) (1406439343.664741, 1406439419.592154, 'b-n', 11000, '(-0.045+0.45j)', 7, [30962137, 396980025, 854396551, 1134174945, 853437138, 396202636, 30643926])
(27/294: 62.778s) (1406439420.287868, 1406439483.06595, 'c-n', 11000, '(-0.045+0.45j)', 7, [528119782, 528126679, 528126607, 528119844, 528106868, 528090927, 528106651])
(28/294: 73.182s) (1406439483.568793, 1406439556.750723, 'b-b', 11000, '(-0.045+0.45j)', 7, [6787090, 446295211, 952251426, 887842857, 952018789, 444850328, 6751657])
(29/294: 69.557s) (1406439557.260535, 1406439626.817696, 'c-c', 11000, '(-0.045+0.45j)', 7, [528123032, 528126337, 528126317, 528123177, 528103166, 528092169, 528103160])
(30/294: 71.463s) (1406439627.654691, 1406439699.118083, 'b-n', 11000, '(-0.045+0.45j)', 8, [12248655, 276845162, 640458941, 918456756, 918692944, 640651609, 277129577, 12313714])
(31/294: 61.419s) (1406439701.319855, 1406439762.738971, 'c-n', 11000, '(-0.045+0.45j)', 8, [462101201, 462097709, 462096354, 462104604, 462098891, 462104601, 462096218, 462097780])
(32/294: 74.349s) (1406439763.250448, 1406439837.599634, 'b-b', 11000, '(-0.045+0.45j)', 8, [83533222, 572258118, 1008374595, 183843579, 183621542, 1008687128, 572715690, 83763484])
(33/294: 63.601s) (1406439838.165366, 1406439901.7661, 'c-c', 11000, '(-0.045+0.45j)', 8, [462098824, 462095653, 462102609, 462095578, 462095605, 462104487, 462099901, 462104701])
(34/294: 60.616s) (1406439903.164532, 1406439963.780276, 'b-n', 11000, '(-0.045+0.45j)', 9, [4925883, 183016549, 499111699, 715249476, 896278422, 714337094, 497544322, 181478870, 4855043])
(35/294: 60.553s) (1406439964.579545, 1406440025.132668, 'c-n', 11000, '(-0.045+0.45j)', 9, [410741906, 410737132, 410741974, 410750876, 410769531, 410767668, 410767726, 410769606, 410750939])
(36/294: 69.629s) (1406440026.029944, 1406440095.659217, 'b-b', 11000, '(-0.045+0.45j)', 9, [8579385, 427601933, 249814502, 490058575, 1344688900, 490058599, 249814296, 427601794, 8579374])
(37/294: 51.912s) (1406440096.418413, 1406440148.330354, 'c-c', 11000, '(-0.045+0.45j)', 9, [410751485, 410754468, 410754555, 410756806, 410762587, 410756876, 410754556, 410754540, 410751485])
(38/294: 55.903s) (1406440149.653172, 1406440205.556156, 'b-n', 11000, '(-0.045+0.45j)', 10, [3770571, 123837603, 366462193, 586688823, 767250324, 767430934, 586813991, 366725810, 124038227, 3778882])
(39/294: 56.107s) (1406440206.719589, 1406440262.826881, 'c-n', 11000, '(-0.045+0.45j)', 10, [369682922, 369681132, 369674794, 369676626, 369680155, 369689053, 369680152, 369676692, 369674641, 369681191])
(40/294: 67.764s) (1406440263.756064, 1406440331.519789, 'b-b', 11000, '(-0.045+0.45j)', 10, [30438885, 350901601, 628804333, 807360406, 30504289, 30438040, 807340437, 629161704, 351259940, 30587723])
(41/294: 66.957s) (1406440332.213713, 1406440399.170588, 'c-c', 11000, '(-0.045+0.45j)', 10, [369679372, 369678018, 369678558, 369678548, 369678168, 369685549, 369677164, 369682447, 369682372, 369677162])
(42/294: 53.369s) (1406440400.687472, 1406440454.056887, 'b-n', 11000, '(-0.045+0.45j)', 11, [3013488, 85985841, 287115608, 486175090, 614021971, 743483469, 614182540, 486351466, 287295517, 86151974, 3020394])
(43/294: 54.508s) (1406440455.090445, 1406440509.598092, 'c-n', 11000, '(-0.045+0.45j)', 11, [336084165, 336088852, 336076254, 336070636, 336060695, 336060122, 336060062, 336060692, 336070703, 336076330, 336088847])
(44/294: 54.738s) (1406440510.267967, 1406440565.006075, 'b-b', 11000, '(-0.045+0.45j)', 11, [2319295, 29929569, 350035841, 581368098, 597416456, 574113506, 597308476, 581543631, 350408188, 30030022, 2324276])
(45/294: 57.531s) (1406440565.767961, 1406440623.298978, 'c-c', 11000, '(-0.045+0.45j)', 11, [336079361, 336072968, 336073255, 336076735, 336074614, 336061650, 336061639, 336074334, 336076731, 336073185, 336072886])
(46/294: 64.956s) (1406440624.940203, 1406440689.896576, 'b-n', 11000, '(-0.045+0.45j)', 12, [2469761, 60665409, 226308308, 397081626, 508838127, 654204336, 653761606, 508531033, 396542913, 225680672, 60262263, 2451304])
(47/294: 54.343s) (1406440690.601154, 1406440744.944298, 'c-n', 11000, '(-0.045+0.45j)', 12, [308066895, 308071799, 308063097, 308062891, 308066229, 308062823, 308063034, 308071800, 308066968, 308067688, 308066441, 308067693])
(48/294: 60.943s) (1406440745.737092, 1406440806.679765, 'b-b', 11000, '(-0.045+0.45j)', 12, [1749744, 112879631, 504299595, 67066850, 198491188, 963728487, 963798300, 198788099, 66913832, 504337128, 112992390, 1752114])
(49/294: 50.527s) (1406440807.33579, 1406440857.862712, 'c-c', 11000, '(-0.045+0.45j)', 12, [308063205, 308067084, 308065765, 308064454, 308068091, 308068674, 308070906, 308068598, 308063133, 308064382, 308065839, 308067227])
(50/294: 349.402s) (1406440962.815184, 1406441312.21672, 'numpy', 11000, '(-0.045+0.45j)', 1, [3696797358])
Traceback (most recent call last):
File "benchmark_julia.py", line 362, in <module>
cli(sys.argv)
File "benchmark_julia.py", line 357, in cli
kernel=fn_from_kernel[args.kernel])
File "benchmark_julia.py", line 289, in do_julia_runs
print(prog_fmt.format(n, n_runs, result[1] - result[0]), result)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/contextlib.py", line 154, in __exit__
self.thing.close()
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 705, in close
self.cleanup()
File "/home/rgrant/development/distarray/distarray/dist/context.py", line 702, in cleanup
prefix=self.context_key)
File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 21, in cleanup
view.apply_sync(_cleanup, module_name, prefix)
File "<string>", line 2, in apply_sync
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 75, in spin_after
ret = f(self, *args, **kwargs)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 247, in apply_sync
return self._really_apply(f, args, kwargs, block=True)
File "<string>", line 2, in _really_apply
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 66, in sync_results
ret = f(self, *args, **kwargs)
File "<string>", line 2, in _really_apply
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 51, in save_ids
ret = f(self, *args, **kwargs)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/view.py", line 577, in _really_apply
return ar.get()
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/asyncresult.py", line 118, in get
raise self._exception
IPython.parallel.error.CompositeError: one or more exceptions from call to method: _cleanup
[Engine Exception]EngineError: Engine 0 died while running task 'f167978c-bab6-433d-a342-95cb22986faa'
[Engine Exception]EngineError: Engine 1 died while running task '23c16002-0e2e-49b5-bc0f-3fcbc99b85bb'
[Engine Exception]EngineError: Engine 2 died while running task '15c2bc93-e5c9-4bd1-9dda-639a53ba4ccb'
[Engine Exception]EngineError: Engine 3 died while running task '4d92d260-c7da-4d41-8e13-ebec83aaabe4'
.... 8 more exceptions ...
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 30, in cleanup_all
v = c[:]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
return self.direct_view(key)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
targets = self._build_targets(targets)[1]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
raise error.NoEnginesRegistered("Can't build targets without any engines")
NoEnginesRegistered: Can't build targets without any engines
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 71, in clear_all
v = c[:]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
return self.direct_view(key)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
targets = self._build_targets(targets)[1]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
raise error.NoEnginesRegistered("Can't build targets without any engines")
NoEnginesRegistered: Can't build targets without any engines
Error in sys.exitfunc:
Traceback (most recent call last):
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/rgrant/development/distarray/distarray/dist/ipython_cleanup.py", line 71, in clear_all
v = c[:]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 923, in __getitem__
return self.direct_view(key)
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 1363, in direct_view
targets = self._build_targets(targets)[1]
File "/home/rgrant/miniconda/envs/test-distarray-27/lib/python2.7/site-packages/IPython/parallel/client/client.py", line 567, in _build_targets
raise error.NoEnginesRegistered("Can't build targets without any engines")
IPython.parallel.error.NoEnginesRegistered: Can't build targets without any engines