pickingproblems

Questions to Answer When Picking Problems

George Heilmeier was a pioneer in developing LCD technology, and was Director of DARPA. As part of his position, he crafted a set of questions known as the "Heilmeier Catechism" to help Agency officials think through and evaluate proposed research programs.

• What are you trying to do? Articulate your objectives using absolutely no jargon.
• How is it done today, and what are the limits of current practice?
• What is new in your approach and why do you think it will be successful?
• Who cares? If you are successful, what difference will it make?
• What are the risks?
• How much will it cost?
• How long will it take?
• What are the mid-term and final “exams” to check for success?

An example

All of the following questions eventually need to be answered in a publication, and ideally before fully jumping into a project. Often you want to run small experiments in order to answer these questions.

All credit goes to remcochang@tufts

• In once sentence, what new thing do you want to show/achieve?

    My new threading library will make systems run faster.
  

• Is this an application, a simulation, a prototype, etc?

    application/library
  

• Who are the users whose lives would be improved by the research?

  • What causes their frustrations in the current state of the art?

  • If your project succeeds, how does that change their lives?

  • If your project is not done, what happens?

      People that user multi-threaded libraries.  Developers to a lesser extent
    
      * programs are too slow.  Some take 10 whole seconds to run!
      * programs will be fast.  completely changes how people interact with programs
      * lives are wasted waiting for slow programs
    

• If the project works out in the best case, what results does that entail?

    Multi-threaded programs run 30x faster
  

• What steps are needed to make this happen?

    1. Identify key bottlenecks in existing programs.  
    2. Use technique XXX to remove the bottlenecks.
    3. Write a library.  
    4. Modify existing programs to use library and run experiments_
  

• What risks could derail the project?

  • Major: would kill the project:

      Technique XXX doesn't work.  (hint: better verify this isn't the case!)
    

  • Minor: slow the project down:

      Programs are super complicated, and takes longer to switch to my library
    

• List the top 5 papers related to this project

      (Consider previous attempts, alternatives, assumptions you rely on, etc)
  
      1. Library Y[1] proposed technique Y to run programs faster.  
         It works really well, however they don't work 
         for mail servers, web servers and databases, which are important 
         cases that affect the users described above.
  
      2. Project W[2] say that multi-threading is not needed if developers use a new chip.  
         However the chip doesn't actually exist yet
  
      3. Developers speed up their programs using D, E, F.[3,4,5]  
         Turns out they are special cases of our library
  
      4. Projects L,K[6,7]  used variations of technique XXX in other domains.
         Look, they worked!  We needed to extend these ideas to make it work for our library.
  

• How do you know if the project succeeded?

  • What do you need to be measuring (aka the x and y axes of a graph)

  • How much do these metrics need to change before "success"?

      The mail, webserver and database programs run at least 2x faster when using our library_ 
      We will measure latency, development costs