Integrate with the IPython notebook

[khinsen]

IPython notebooks look like a great approach for combining the outermost layer of computation with a presentation of the methods and results.

Implementation outline:

• Make a new top-level group "notebooks" for notebook files.
• Add a new subcommand to aptool that starts an IPython notebook server.
• The notebook server runs a kernel that has access to the ActivePaper's code.

Disadvantage: a big dependency with non-trivial installation. Precautions should be taken to make the resulting ActivePapers useable without IPython, e.g. by storing a static HTML view of the notebooks along with the original files.

[khinsen]

After a first look at the IPython notebook architecture, the first complication shows up. When N notebooks are open, there are N+1 independent active processes: N kernels, one for each notebook, plus the notebook server. Each of these processes needs write access to the ActivePaper, but at any moment in time, only one process can have the file open for writing.

Consequences:

• The ActivePaper must be opened and closed frequently.
• Write operations need to be synchronized by file locking.
• Write access must pass from the notebook server to a kernel whenever a cell is executed, and back to the server when cell execution terminates. This requires some hook into the communication between the server and the kernels.

[Carreau]

Hi,

I would suggest a custom "proxy" for the kernel that intercept/buffer the ZMQ communications.
you could set the kernel_cmd of NotebookManager to yourproxy requested-kernel other-params.
Then you will have access on the fly at the execution_requests and reply which would let you know when to give write access to the server/kernel.

This protocol is the more stable we have, but will probably have some addition/modification in 3.0 (summer/fall).

You can also define your own backend for the notebook-server(see bookstore, so I suppose this + fuse could allow ou to have a single process writing and manage the lock by yourself.

[khinsen]

Thanks, that helps to get started!

Changing kernel_cmd is easy enough (and works fine), but it seems that I'd rather keep IPKernelApp and change kernel_class to something of my own. However, that has no effect whatsoever. In fact, I can even set IPKernelApp.kernel_class = 42 and everything works fine as if nothing has happened. Is this setting treated specially somehow?

[Carreau]

How are you changing it ? In config with c.IPKernelApp... ? or on the command line ?

If the second as the kernel is a subprocess it might not get the config and use default config.

[khinsen]

I am changing it in ipython_notebook_config.py:

c = get_config()
c.IPKernelApp.kernel_class = 'activepapers.ipkernel.ActivePapersKernel'

I can set it to whatever I want, it makes no difference. Is notebook_config perhaps not the right file to use?

[Carreau]

ipython notebook config might apply only to server as kernel can be started to any frontend.
Try setting it in ipython_config.py

[khinsen]

That works! Except that now I get my personal kernel all the time, whereas I plan to modify it in such a way that it requires the notebook server as a communication partner.

After some more experimentation, I found something that works but maye relies on undocumented features:

1. I create my notebook app in a small Python script:

from IPython.html.notebookapp import NotebookApp
from IPython.kernel import make_ipkernel_cmd

kernel_cmd = make_ipkernel_cmd('from activepapers.ipkernel import main; main()')

app = NotebookApp()
app.config.KernelManager.kernel_cmd = kernel_cmd
app.initialize()
app.start()

2. My kernel_cmd creates an IPKernelApp and changes its kernel_class:

def main():
    app = IPKernelApp.instance()
    app.config.IPKernelApp.kernel_class = 'activepapers.ipkernel.ActivePapersKernel'
    app.initialize()
    app.start()

Is this way to change the configurations from Python code safe?

[Carreau]

Supposing app.initialize does not load config that have kernel_class, I suppose.

But you fall into the case that IPython 2.0 and below do not have the ability to "choose" a kernel form the frontend.
We will add that for 3.0, I suppose you when just "register" your kernel(s) I'll try to ping you if there is PR about multikernel/ kernel choosing from the forntend.

The way I see it, I would write a kernel wrapper, that would make active-paper write regardless on the kernel type right ? than you would support Julia , Ruby, Go, Haskell... out of box.

Well haskell has no side effect so it might not be a Good example :-)

[khinsen]

A frontend-specific kernel is indeed exactly what I need, so I am looking forward to the 3.0 developments.

As for configuration, I do have to make some assumptions about users being reasonable in their config files, as obviously they can break my setup by redefining low-level stuff. I'll worry about such details when I have something that works :-)

Supporting other languages than Python is currently not on my agenda. Of the languages you list, only Julia has a usable HDF5 interface at the moment, and none has an interface to the ActivePapers machinery.

[khinsen]

For those interested in following development of IPython notebook integration, it happens on this branch in my personal fork: https://github.com/khinsen/activepapers-python/tree/ipython
Note: watch out for rebasing.

The main problem is the use of multiple Python processes by IPython (one for the server/notebook manager plus one per kernel/open notebook), which conflicts with the HDF5-derived limitation that only one process at a time can open a file for writing. I am looking into the possibility of having multiple kernels running in a single process, with separate namespaces. If that is not possible, users will have to live with a limitation to one open notebook at a time. For the notebook server, I am considering having it write changes to a temporary file which is then copied to the ActivePaper by the kernel at regular intervals. The main problem with that approach is the risk of missing changes.

In view of this enormous amount of incidental complexity, I am also considering to abandon the use of the IPython framework, and making my own single-process notebook interface with a compatible format, re-using only individual modules from IPython. That's a lot of work as well, but may result in something easier to maintain in the long run.

[khinsen]

One possible solution: make a separate ActivePaper for each notebook, creating links to everything in the original ActivePaper to make the data and code in it accessible (read-only) from the notebook. Any dataset produced by the notebook goes into the notebook-specific file. For publication, any number of notebook files can be merged with the original ActivePaper. It would still be possible to re-run a notebook completely from such a merged file, but only as a whole, not interactively.

The main downsides I can think of:

• The merge step must be run explicitly by the user, creating a source of mistakes.
• If a notebook tries to overwrite a dataset in the original ActivePaper, or if two notebooks try to write to the same dataset, the conflict is detected only at the merge phase. A partial test can be implemented at notebook runtime, but the specific case of two notebooks run in parallel and writing to the same dataset cannot be detected at this stage.