Building a Chumby-style feed reader using SJS

Stratified JavaScript (SJS) might still be work in progress, but that doesn't mean it can't be used for building some (semi-)useful things.

Consider e.g. the 'FeedChumby' RSS feed reader which automatically cycles through the posts of some popular RSS feeds, 'slide-show' style - similar to the way in which a Chumby cycles through its widget stream.

In this blog post I will dissect the code for this webapp, tutorial-style.

Accessing feeds

At its core, FeedChumby uses the Google AJAX Feeds API which allows us to retrieve RSS feeds without having to worry about any crossdomain issues and without requiring any server support from our side.

SJS gives us access to this API through the unassuming function

  Ajax.Google.Feeds.loadFeed(url);

which returns a JSON results object as described in the Google documentation.

Because we're using SJS, we can call this function synchronously - no callback or other asynchronous construct needed. All the asynchronous action is automatically choreographed behind the scenes: loading the Google API if it hasn't been loaded yet, making sure the 'feeds' module is loaded, and finally loading the actual feed itself. While all of this is going on, the browser stays perfectly responsive.

Coupled with SJS's 'Ajax.DOM.waitForEvent' library function (which - as one would expect from the name - waits for a particular DOM event), loading and displaying a feed in response to a button click is then as easy as this:

  Ajax.DOM.waitForEvent("click", button);
  var result = Ajax.Google.Feeds.loadFeed(feed_url);
  renderResult(result);

Here is a complete example: google-feeds.html

Basic program structure

By sticking 'loadFeed' into a loop, iterating over a few different feeds and over individual feed posts, and pausing for a few seconds after each post is displayed (using SJS's "hold()" primitive), we've got the basic structure for our Chumby-style feed reader - again no asynchronous constructs (like e.g. setTimeout) needed:

  var feeds = [ "...slashdot...", "...digg...", "...elreg...", ... ];

  while (true) {
    for (var i=0; i<feeds.length; ++i) {
      var result = Ajax.Google.Feeds.loadFeed(feed[i]);
      for (var j=0; j<result.feed.entries.length; ++j) {
        renderFeedEntry(result.feed.entries[j]);
        
        hold(10000); // <-- wait 10s before continuing with next post
      }
    }
  }

Let's also add a button that allows the user to shortcut through the 10s waiting time. The easiest way to provide this sort of functionality is by using SJS's '@' combinator (as described in an earlier blog post, "A @ B" executes both A and B in parallel, and returns as soon as either of them have finished evaluating; at the same time cancelling the still pending expression):

  for (...) {
    ...
    hold(10000) @ Ajax.DOM.waitForEvent("click", nextButton);
  }

As a little refinement, we can replace the 'hold(1000)' by code that will provide the user with some UI feedback during the waiting period:

  function countdown() {
    for (var i=0; i>0; --i) {
      counter.innerHTML = i;
      hold(1000);
    }
  }

  ...

  for (...) {
    ...
    countdown() @ Ajax.DOM.waitForEvent("click", nextButton);
  }

Here is a complete example with the functionality so far: feedchumby.html

Adding 'pause' functionality

Again using '@' and 'Ajax.DOM.waitForEvent()', we can modify our countdown() function to stop counting down while in a paused state:

  var paused = false;

  function countdown() {
    var i=10;
    while (i>0) {
      counter.innerHTML = i;
      if (paused) {
        pauseButton.innerHTML = "Play";
        Ajax.DOM.waitForEvent("click", pauseButton);
        paused = false;
        pauseButton.innerHTML = "Pause";
      }
      (hold(1000), --i) @ 
      (Ajax.DOM.waitForEvent("click", pauseButton), paused=true);
    }
  }

Note the '@' expression here. It combines the two simpler expressions "(hold(1000), --i)", which waits for 1s and then decrements 'i', and "(Ajax.DOM.waitForEvent("click", pauseButton), paused=true)", which waits for a click on 'pauseButton' and then sets the variable 'paused' to true. (In both 'normal' JavaScript and also SJS, for two expressions A and B, "A,B" is an expression that means "execute first A and then B".)

Here is the example with the pause functionality rolled in: feedchumby2.html

Adding birectional navigation

To add functionality that allows the user to navigate back to the previous post, we need to change our loop logic a little bit to account for the situtation where we're moving past the beginning of a feed to the previous feed. The gory details are here: feedchumby3.html.

Note the '@' expression in the main loop:

  (countdown(), ++j) @ 
  (Ajax.DOM.waitForEvent("click", "next"), ++j) @
  (Ajax.DOM.waitForEvent("click", "previous"), --j);

Previously we only waited for the countdown to finish or for the user to click on the 'next' button. Now we're also waiting for clicks on the 'previous' button, and we're incrementing/decrementing variable 'j' to communicate the direction of navigation.

More embellishments

The absence of any asynchronous constructs (callbacks, setTimeouts, ...) makes the code quite modular. It's easy to add little features here and there without having to take a "global view" of the application, and the '@' combinator relieves us from a great deal of manual bookkeeping.

The final version of the FeedChumby adds a couple more small touches, like e.g. an animated pause button, and it contains some CSS to make the application more presentable: feedchumby4.html

I think FeedChumby is a pretty good example of the power of SJS for writing Ajax-style applications. The real win, however, will only show in big 'mashup' style applications that access several different APIs concurrently. Expect to see more demos coming along with APIs such as the Google Translation API, Google Contacts, Facebook Connect, and others.

Stratified JavaScript and Multitasking

In response to my earlier blog post on Stratified JavaScript, Jonathan Cook asked an interesting question:

"[...] I am very interested in using SJS to eliminate the blocking
nature of heavy computation in nested loops.

A hold(0) would seem to do this, or perhaps a different method for
this purpose would be appropriate?"

The short answer is, you don't need to call hold() or do anything special at all. Just code your computation as you would in 'normal' JavaScript, and SJS will take care of the rest; the UI will automatically stay responsive while your computation is chugging along.

Now for the slightly longer answer.

The SJS runtime contains a scheduler which preemptively schedules the different strands of execution ('strata') of an SJS program. This scheduler automatically yields control to the browser every so often, and it ensures that individual strata get a fair timeslice.

In other words, SJS implements a form of preemptive multitasking, rather than cooperative multitasking. There is no need to manually yield control like you would in e.g. Narrative JavaScript/Strands; SJS automatically does it in the background.

Here's an example of a heavy computation in SJS; a program to calculate a few hundred digits of Pi:
pi.html

There are a couple of noteworthy things about this program:

Firstly, if you look at the source, there is no hold() or any other special SJS primitive. It is just 'straight' JS code, which is being preprocessed through the SJS compiler by virtue of sitting in a script tag with type "text/sjs".

Secondly, it runs painfully slow. This is a consequence of the fact that the SJS compiler doesn't perform any optimizations at all at this point; every JavaScript function call or operator application is converted into a potential yielding point, adding huge overhead to programs which are heavy on mathematical calculations.

This situation will improve dramatically when we get around to making the compiler smarter. A similar optimization in Oni resulted in a 2 orders of magnitude improvement!

In the meanwhile there is a workaround though: SJS code can freely call 'normal' JavaScript code, so we can just move critical inner loops into 'normal' JavaScript, as in this modified program:
pi-faster.html

You'll notice that this program runs a lot faster, while not really adding much complexity - just one inner loop has been moved to 'straight' JavaScript.

Just to prove that the preemptive scheduler works as advertised, here are another couple of examples which add some SJS-specific features.

Firstly, this program adds an animation running in parallel to the Pi calculation:
pi-and-animation.html

Secondly, and finally, this program adds a 'UI' for turning the animation on and off:
pi-and-animation-and-events.html

Take a look at the animation loop:

  function animateLoop() {
    var elem = document.getElementById("toggle_animation");
    while (true) {
      waitForEvent("click", elem);
      elem.innerHTML = "Stop animation";
      animate() @ waitForEvent("click", elem);
      elem.innerHTML = "Restart animation";
    }
  }

Isn't it beautiful? :-)

Introducing Stratified JavaScript

Here's a sneak preview of a project I've been working on in my spare time for a while, and which is nearing its first release:

'Stratified JavaScript' ('SJS') is a cross-browser extension to JavaScript which adds some concurrency features to the language. Without going into too much detail, the main features include:

(1) The ability to pause execution

The function

  hold(time_ms);

pauses execution of a piece of code for time_ms milliseconds. E.g.:

  var elem = document.getElementById("animated_element");
  var x = 0;
  while (true) {
    elem.style.left = x; 
    x = (x + 10) % 200;
    hold(100);
  }

(full sample: animate.html)

Note that 'hold' doesn't busy wait. The browser's UI stays fully functionally while this code executes.

(2) Fork-Join Parallelism

In conventional JavaScript, arguments to function calls are evaluated sequentially. In Stratified JavaScript, they are evaluated concurrently:

  function animate(elemname, duration_ms, step_ms) {
    var elem = document.getElementById(elemname);
    var start_time = new Date();
    var x = 0;
    do {
      elem.style.left = x;
      x = (x + 10) % 200;
      hold(step_ms);
    } while (duration_ms > new Date() - start_time);
  }

  function par(a, b) {
    alert("all done");
  }

  par(animate("animated_element_1", 10000, 100),
      animate("animated_element_2", 7000, 80));

(full sample: forkjoin.html)

Here, the two 'animate' calls are being executed concurrently. Once they both return, the body of 'par' gets called.

(3) Exploratory parallelism

Many concurrency problems fit into the pattern

"Explore options a,b,c,... concurrently. Once one of them yields a satisfactory result, return it and abort exploration of the remaining options."

Exploratory parallelism in Stratified JavaScript is embodied by the '@' ('alt', 'alternatives') operator:

  animate("animated_element_1", 10000, 100) @ 
  animate("animated_element_2", 7000, 80);

  alert("all done");

(full sample: alt.html)

This code pops up 'all done' after 7s (after the animation of animated_element_2 has finished). At the same time, the animation of animated_element_1 will be aborted.

In many ways, exploratory parallelism forms the heart of Stratified JavaScript. It is applicable to many situations which are very cumbersome to express in conventional thread-based parallelism.

(4) Suspend/Resume

Most JavaScript programs live by asynchronous events, e.g. events generated by button clicks, or asynchronous XMLHttpRequests. Stratified JavaScript contains a generic mechanism for converting these asynchronous constructs into synchronous ones:

   suspend {
     ... resume() ...
   }
   retract {
     ...
   }
   finally {
     ...
   }

This will be executed by first evaluating the code in the 'suspend' block and then suspending execution. Execution will resume when the 'resume' function defined in the suspend block is called. The optional 'retract' block will be executed if the current code is aborted before 'resume' was called. The optional 'finally' block will be executed in either case; if execution was resumed, or if execution was aborted.

E.g., this is how an event listener could be 'synchronized':

  function waitForEvent(event, elem) {
    suspend {
      var rv;
      var listener_func = function(e) {
        rv = e;
        resume();
      };
      elem.addEventListener(event, listener_func, false);
    }
    finally {
      elem.removeEventListener(event, listener_func, false);
    }
    return rv;
  }

With this function we can now e.g. wait for a button click:

  while (true) {
    waitForEvent("click", document.getElementById('button1'));
    alert("You clicked the button");
  }

(full sample: suspend.html)

Or something more complicated:

  function dump(message) {
    document.getElementById("output").innerHTML = message;
  }

  while (true) {
    var e = waitForEvent("click", document.getElementById('button1')) @
            waitForEvent("click", document.getElementById('button2')) @
            hold(5000);
    if (e)
      dump("You clicked button '"+e.target.id+"'");
    else
      dump("Click a button already!");
  }

(full sample: suspend2.html)

If all of this sounds similar to Oni, that's because it is. Under the hood, Stratified JavaScript is executed by a runtime based on Oni. Everything that can be expressed in Stratified JavaScript can be written directly in Oni without going through any compilation stage. The advantage of Stratified JavaScript is that it doesn't have the awkward functional syntax of Oni.

Ubiquity, Oni, and Composability

Ubiquity and Composability

One case where the limitations of conventional asynchronous code bubble to the surface is in systems like Ubiquity. E.g. Ubiquity has a 'google' command and a 'translate' command. I can instruct Ubiquity to google for something:

google foo

and I can get Ubiquity to translate something:

translate The quick brown fox to German

but I cannot instruct it to give me list of translated google results:

translate (google foo) to German  // doesn't work

Or email me the resulting list:

email(translate (google foo) to German) // doesn't work

Or email me a list of laptops on ebay below some price:

email(filter(price<500dollars, find_on_ebay(macbook)) // doesn't work

(Ok, excuse the syntax for these examples; in 'real-world' Ubiquity you would probably want this to be in a more natural language style, but I hope I'm getting across the idea of composability. Or rather: lack thereof.)

Oni and Composability

So how does Oni relate to this? Oni is a browser-based "embedded structured concurrency framework". It allows you to write asynchronous code as if it was synchronous, adding back the kind-of composibility that is lost when juggling concurrent strands of execution (such as e.g. pending XMLHttpRequests) with 'conventional' sequential languages.

Oni has recently gained a small AJAX library with some bindings to the Google AJAX APIs. With this new functionality we can write some semi-useful Oni programs which nicely demonstrate the type of composability that Oni provides.

One of the new Oni functions is Google.Feeds.Load(url), which retrieves a JSON object of the RSS feed at 'url'.

With this function it is easy to write an Oni program that e.g. displays the most recent item posted to planet.mozilla.org:

Display(GetMostRecentPost("http://planet.mozilla.org/atom.xml"))

(full sample: composability1.html)

where Display, and GetMostRecentPost are implemented as follows:

var Display = SLift(function(html) { 
                      document.getElementById('output').innerHTML += html; 
                    });

'SLift' is an operator that 'lifts' a conventional sequential JavaScript function into Oni.

var GetMostRecentPost = Defun(['url'], 
                              ObjMem(Google.Feeds.Load(Get('url')),
                                     'entries', 0, 'content'));

'GetMostPost' is a new Oni function which uses 'Google.Feeds.Load' to load the feed, and then traverses the returned feed JSON object using ObjMem, to finally return the html content of the current first post.

So far so good, but this is hardly exciting. Let's add another function into the mix: Google.Language.Translate(text, srclang, destlang), which returns a translation of text (assumed to be in language 'srclang') to language 'destlang'. The object returned by this function is a JSON 'translation result' object of which we only want the 'translation' member, so let's wrap it into a 'Translate' function which extracts just this member:

var Translate = 
  Defun(['txt', 'lang'],
        ObjMem(Google.Language.Translate(Get('txt'), 'en', Get('lang')), 
               'translation'));

Armed with this new function, we can now do something a little more interesting, e.g. retrieve a feed entry and translate it into German:

Display(Translate(GetMostRecentPost("http://planet.mozilla.org/atom.xml"),
                  'de'))

(full sample: composability2.html)

Or retrieve a post and display it in English, German, Spanish and Japanese:

Let({post:GetMostRecentPost("http://planet.mozilla.org/atom.xml")},
    Par(Display(Get('post')),
        Display(Translate(Get('post'), 'de')),
        Display(Translate(Get('post'), 'es')),
        Display(Translate(Get('post'), 'ja'))))

(full sample: composability3.html)

Here, 'Let' introduces a new variable ('post'). 'Par' executes its subexpressions in parallel, so the ordering of what's displayed on screen will depend on the order in which the translation requests return.

Here's another example: Fire off requests to get the most recent post from slashdot and the most recent post from planet mozilla and display a translated version of the first one that comes in:

Display(Translate(Alt(GetMostRecentPost("http://rss.slashdot.org/..."),
                      GetMostRecentPost("http://planet.mozilla.org/...")),
                  'de'))

(full sample: composibility4.html)

Here, 'Alt' is an Oni operator that executes its subexpressions in parallel and returns the first one that returns, at the same time cancelling the other one.

And for a final example, let's add a timeout to the previous example. If we don't get a translation of a planet or slashdot post within 1 second, cancel any pending requests and display a message instead:

Display(Alt(Translate(Alt(GetMostRecentPost("http://rss.slashdot.org/..."),
                          GetMostRecentPost("http://planet.mozilla.org/...")),
                      'de'),
            Delay(1000, "Sites timed out")))

(full sample: composibility5.html)

So what's the big deal?

What these examples illustrate is composability, something that the conventional way of dealing with concurrency lacks. The 'standard' way of coding the examples above would be by using callback functions. This works well enough for simple cases, but as programs become more complex you soon enough end up in 'callback hell'. In Oni (and similar systems such as Orc or Arrowlets), we can build more complex programs out of simpler ones in a modular, structured fashion.

So what's the relationship to Ubiquity? Oni is a low-level framework layered on top of JavaScript. It is not, and does not aspire to be a human interface in the way that Ubiquity is. The relation between the two is only tangential: I believe that Ubiquity could become much more expressive if layered on top of a system like Oni.

If this got you interested, and you'd like to find out more about Oni, please visit the project homepage at http://www.croczilla.com/oni.