Collaboration: Theory and Practice

So much has happened since we started this project. We didn’t really know what we’d end up doing – we just had a vision and a bunch of ideas. At this point we realized how naive our approach was – we needed foundations much deeper in order to achieve our vision.

Anyways, here is a short video about what we’ve been doing. Enjoy.

(defn add-to-class [partition class el]
  (conj (disj partition class) (conj class el)))

(defn containing-class [partition el]
  (first (for [class partition :when (some #{el} class)] class)))

(defn link-elements-in-partition [partition el1 el2]
  (let [c1 (containing-class partition el1)
        c2 (containing-class partition el2)]
    (cond

      (and (not c1) (not c2)) ; neither are in any class (we add it to c1)
      (conj partition #{el1 el2})

      (and c1 (not c2)) ; el1 was found in a class (c1) while el2 was not (we add it to c2)
      (add-to-class partition c1 el2)

      (and c2 (not c1)) ; el2 was found in a class (c2) while el1 was not
      (add-to-class partition c2 el1)

      (and (and c1 c2) (not= c2 c2)) ; both were found in different classes (we join them)
      (conj (disj partition c1 c2) (union c1 c2))

      :else
      partition)))

TRACE t32800: (link-elements-in-partition #{#{1 2} #{3 4 5}} 3 2)
TRACE t32801: | (containing-class #{#{1 2} #{3 4 5}} 3)
TRACE t32801: | => #{3 4 5}
TRACE t32802: | (containing-class #{#{1 2} #{3 4 5}} 2)
TRACE t32802: | => #{1 2}
TRACE t32800: => #{#{1 2} #{3 4 5}}

(start-complete-tracing)
(link-elements-in-partition #{#{1 2} #{3 4 5}} 3 2)
(end-complete-tracing-and-print-org-mode)

**** (and (and c1 c2) (not= c2 c2))
**** `=> Result:
false
***** (and c1 c2)
***** `=> Result:
#{1 2}
****** c1
****** `=> Result:
#{3 4 5}
****** c2
****** `=> Result:
#{1 2}
***** (not= c2 c2)
***** `=> Result:
false
****** c2
****** `=> Result:
#{1 2}
****** c2
****** `=> Result:
#{1 2}

This post has a follow-up.

We are all familiar with the following scenario – You write some code. Everything is working. You change your code just a little bit…

Incidental logging and step-by-step debugging are basically the same thing. You make your way through your code in a divide-and-conquer search for the cause of an unexpected result. You add a log here, you debug a function there, you step-into-outta-over-and-out. And then you finally find it – the function that returns the wrong value. And then, hopefully, the line that messes everything up. It took you almost forever and you know that with the next bug everything will happen all over again. But you always have a feeling that there is some reason guiding you through this weird process. And it’s true – Logging is clearly a job for a computer. So we asked ourselves – why not automatically log all arguments and return-values for any function call?

If you were to try implementing this in other languages you would find out you need to pursue one of two approaches, both fundamentally flawed. One would be to preprocess your code and inject logs. Another approach would be to implement some hideous reflection mechanism to inject the logs in the right places. The first approach would require heavy coding and either text-parsing or some use of a code analysis library, the second approach would require compiling code at runtime. Luckily for us, a preprocessed code-walking in Lisp is as simple as writing a function that iterates a list. Well, It is iterating a list!. Moreover, what would be the right places to inject logs at? Well, if you write in functional style, you really know for certain that you only need to log the arguments and return value of every function (or a sub-special-form of a function). In a non-functional programming language (any language that allows modifying variables) this certainty doesn’t exist. Therefore, any attempt to automatically generate all needed logs would require the logging of all variables after each and every single line of code.

In Lisp, a code-walker macro can be written that will iterate a function’s s-expression and “inject” log calls before every sub-expression call. We had some issues with handling Clojure’s special forms like “let” and “for”, but victory was eventually ours. It ended up looking like this.

The question still arises: How will we view such a monstrous amount of logs? We would really want to be able to view it as if its a simulation of the actual run of the program. So we wrote a small Javascript application to display the results of the logging in such a way. And that looks something like this:

With that said, we are aware that as our project will grow in complexity it may not be feasible to continue with this paradigm. However, thanks to Clojure, we believe that we will be able to find ways to maintain some parts of it by improving the logging mechanism to be less verbose and more precise.

Lately we are very interested in Lisp and the immense productivity gain when developing with it, and we are especially enthusiastic about Clojure, a new dialect of Lisp. Therefore we thought of trying to write some Wave robots in Clojure.

Some people have already experimented with writing robots in Clojure (for example here and here). However, these approaches still assume that development is done on Google App Engine (GAE). This means that you have to redeploy each time there is a new version of your robot (contrary to the common Lisp way which is just evaluating code into a remote process, known as a REPL). We also wanted to use an in-memory database instead of dealing with database schemas and other irrelevant issues for prototyping. This can’t be done with GAE since each robot event gets sent to a different machine running the robot code and the memory is not shared.

In order to write in Clojure and use an in-memory database, we had to find a way to write our code outside of GAE. Google currently allows robots to only be run on GAE so the solution was writing a relay robot to manage the communication between the Wave server to our Clojure server. More technically, our relay robot is written in Java and all it does is intercept the JSON objects coming in from the Wave server, post them to a web server running Compojure (a Clojure web server), wait for a response (the JSON object describing the operations to be done on the wave) and pass it back to be processed.

We have been using MooTools to develop our gadgets and that wasn’t causing any problems on top of our emulator. But apparently our gadgets weren’t working on the real Wave sandbox. After a lot of debugging it turned out that Wave’s JSON object conflicts with MooTools’ own JSON object. In order to use develop Wave gadgets using MooTools, use the MooTools Core Builder and remove the JSON component.

If you still want to encode and decode JSON, you can use Wave’s HTML 5-compliant JSON functions (JSON.stringify and JSON.parse instead of MooTools’ JSON.encode and JSON.decode).

Update: Rasmus Fløe brought up a much simpler solution which allows you to still use MooTools’ JSON API. Instead of ommitting MooTools JSON from Core, just extend the JSON object: $extend(JSON, {stringify: JSON.encode, parse: JSON.decode});

Here’s the source code for task list gadget and the original blog post about it.

Brian Flood has just published a robot/gadget combo that displays a map showing the locations of the wave participants using several different geolocation API’s and Google Maps. Here’s some information about robots and gadgets. I don’t have sandbox access so I can’t test it but here are some preliminary comments.

1. This is the first time I’ve seen a robot/gadget combo. The robot, when invited to a wave, automatically embeds a specialized gadget. This seems to be a simpler way to embed specialized content for a typical user than actually installing a gadget, and it allows the robot to place additional structure on the collaboration (for example, permissions). Robots can also access and modify the enclosing wave, whereas gadgets can only access the gadget state. I believe we will see many significant add-ons to Wave being bundled in this way.
2. We are starting to see how the underlying philosophy of Wave is affecting developers. In Wave, people can only access waves if they were invited to join them. This is in stark contrast to the current state of affairs in other social platforms on the Web (Facebook, Twitter). In Brian’s participant map, any participant must opt-in to share their location. We are gracefully entering the next phase of the Web – the phase of true participatory social networks, built on top of Wave.

The lastest version of the gadget emulator now shows the gadget state and supports an actual list of participants and quite a few of the sample gadgets are working now. I am current working with Patrick “p2k” Schneider to try to integrate this into his work on a wave server. There are still a few gadgets that don’t work because of unimplemented APIs on the emulator (_IG_Prefs and OpenSocial), but the current state of the emulator is enough for us to start creating some significant gadgets. If anyone feels that they would like this emulator to support additional APIs, feel free to participant by forking the project on GitHub.

On another note, Vidar’s attempt at an SVG editor has shown how hard it is to manage gadget state information when all you can use for values is strings (no arrays or objects). Vidar made an attempt to solve it in his gadget by encoding an array-like structure as keys of the form “svg_<participant-id>_<index>”. This is a good idea but it is still broken because:

1. You can be logged in as the same user on multiple machines
2. This paradigm would not allow to add elements in the beginning or middle of an array

I am starting to work on a gadget state management framework that should give you the feeling like you’re almost just working with plain Javascript objects and arrays when in fact is uses a crazy encoding scheme to make it work with Wave’s basic gadget state structure without breaking concurrency.

Inspired by Vidar Hokstad’s two-iframes gadget emulator, I have created one that is actually backed up by a real database. It polls every 6 seconds for updates, so it’s not as immediate as Google’s (yet) but that would actually make it easier to discover concurrency problems with gadgets.

Each generated URL points to a specific gadget with a specific instance. You can share these URL’s on multiple machines (or windows if you are testing alone) and they will synchronize data.

http://wave.thewe.net/emulator

Code on GitHub: http://github.com/avital/google-wave-gadget-emulator/tree/master