Alson Kemp

Ran into a nasty issue with URL parsing a while ago in a Rails app.  I was pulling URLs in from Amazon’s shopping APIs and started receiving occasional InvalidURIError exceptions.  Tracked it down to percent signs in the URLs passed from Amazon.  Percent signs are kinda reserved for escaping/encoding some characters in URLs, but, while I’m not sure that the Amazon URLs are truly legal, the parsing function shouldn’t fail on a non-standard usages.  After all, the web is a messy place and failing-safe on parsing messy data seems a better behavior…    Examples:

irb(main):001:0> require 'uri'
=> true
irb(main):002:0> u = URI.parse("http://www.yahoo.com/test?string=thing%ding") # questionable
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/test?string=thing%ding
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:inparse'
 from (irb):3


irb(main):003:0> u = URI.parse("http://www.yahoo.com/abba?test%20text")  # legal usage
=> #<URI::HTTP:0x164cbde URL:http://www.yahoo.com/abba?test%20text>

irb(main):004:0> u = URI.parse("http://www.yahoo.com/abba?test%u2020text") # quasi-legal usage
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/abba?test%u2020text
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:inparse'
 from (irb):5

Enter JRuby

Been thinking about checking out JRuby since I would then have access to the universe of Java libraries.  So Ruby’s URI.parse method is not very robust and I should have caught the exception, tried an alternate parser, etc, but what if I used the Java URL decoder on these URLs?  Works as expected:

irb(main):001:0> include_class "java.net.URLDecoder"
=> ["java.net.URLDecoder"]


irb(main):014:0> u = URL.new("http://www.yahoo.com/test?string=thing%ding")                       
=> #<Java::JavaNet::URL:0x13e4a5a>                                                                
irb(main):015:0> u.getQuery                                                                       
=> "string=thing%ding"                            

irb(main):013:0> u = URL.new("http://www.yahoo.com/test%u2020tring")
=> #<Java::JavaNet::URL:0x3aef16>
irb(main):014:0> u.path
=> "/test%u2020tring"

Lesson

Ruby’s a great little language; Java’s a great big library => JRuby’s a great little language with a great big library.

Also, an interesting article on URL parsing from DaringFireball.

Turbinado has been making solid progress.  See the GitHub repo.

Added a couple of big new features:

HAML and XHTML Templates

While Turbinado used XHTML for templates before, the HSX library (an amazing piece of work) was a real pain to use and threw very strange errors. 

HSX (and its associated libraries) has been torn out of Turbinado and has been replaced by a custom preprocessor called ‘trturbinado’. The preprocessor is run by the webserver over any templates (or views) before they’re compiled and loaded. The preprocessor handles XHTML and a limited version of HAML (to be enhanced).

See the example below. After preprocessing, the ’someHtml’ and ’someHAML’ functions will yield the exact sasme code as the ’someTextXHTML’ function.

Note: I’m not great at writing parsers, so I wouldn’t doubt that there are bugs in the preprocessor!

markup :: VHtml
markup=
  <div>
    <% someTextXHtml %>
    <% someHtml %>
    <% someHAML %>
  </div>
 
-- | These are the raw Turbinado.View.Html style tags
someTextXHtml :: VHtml
someTextXHtml = do s <- getViewDataValue_u "sample_value" :: View String
                   ((tag "div") (
                    ((tag "i") (stringToVHtml s)) +++
                    ((tag "b") (stringToVHtml "some text in Turbinado.View.Html style"))
                    ))
 
someHtml :: VHtml
someHtml = do s <- getViewDataValue_u "sample_value" :: View String
              <div>
                <i><%= s %></i>
                <b>some text in XHtml style</b>
              </div>
 
someHAML :: VHtml
someHAML = do s <- getViewDataValue_u "sample_value" :: View String
              %div
                %i= s
                %b some text in HAML style

FastCGI Support

FastCGI has been an often requested feature, so I rewrote the CGI handling and added FastCGI support, too. Much as with CGI, using FastCGI is very simple. Just tell Apache to use ‘dispatch.fcgi’ to handle requests.

  DocumentRoot /home/alson/projects/turbinado/static
  
    Options +FollowSymLinks +ExecCGI +Includes
    AddHandler cgi-script cgi
    AddHandler fastcgi-script fcgi
    AllowOverride None
    Order allow,deny
    Allow from all
  
  RewriteEngine On
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f
  RewriteRule (.) $1 [L]
  # Use FCGI
  RewriteRule ^(.)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]
  # Use CGI
  # RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]

Performance of Turbinado’s FastCGI interface is pretty good, but is well short of Turbinado’s HTTP interface, so, if you can live without the benefits of FastCGI, HTTP, which is simpler and faster, is the way to go.

Been a long silent few months (blog cliche, anyone?), but Turbinado is cranking along again. Bodo (http://www.naumann.cc/) is using Turbinado for a school project so we’ve been working together to advance Turbinado. So what’s new? Mostly touch ups:

HTTP 4k Support

The HTTP 3000 package seems to be slowing down while HTTP 4000 seems to be taking over.  Makes sense to update Turbinado to the latest version.  Done.

Better Forms Support

Added in support for multipart forms, so now you can upload files to Turbinado.  Note: there is currently no limit on the size of the upload, though.  (See Turbinado.Environment.Files for how to access uploaded files.)

Improved ORM

The PostgreSQL ORM is pretty sweet and now it’s even better.  Improved error handling, quoting of tables names, etc.  Next step is to build the MySQL ORM.

Summary

We’re moving again.  Note: turbinado-website is currently updated, but turbinado isn’t (it’ll be fixed by June 4th).

Would love your feedback on what could be done to improve Turbinado.  Although I made a point of running turbinado.org on Turbinado, it may be time to switch to a CMS that allows people to contribute to the project more easily.

It’s been long enough since Turbinado V0.4 that I figured I’d skip V0.5 and go straight to announcing Turbinado V0.6.  Lots of new excellent features:

• By popular demand, support for CGI serving. Apparently some web hosts don’t support HTTP proxying, so some folks requested CGI support.
• Statically compiled Layouts, Views, Controllers.
• Support for “.format” in routes. If a request path is “/User/List.xml”, then the following View will be called: /App/Views/User/ListXml.hs.
• Lower case paths.
• Support for cookies (see here for examples).
• Encrypted cookie sessions (see here to see how to use them).
• Much easier installs using cabal-install.
• Support for GHC 6.10.  GHC 6.8 is no longer supported.

Turbinado V0.7 will be all about:

• Documentation. (seriously.)
• User authentication.
• Tutorials.

Installation

Installation is pretty painless if you use cabal-install, so make sure that you have cabal-install installed first.  See here: http://hackage.haskell.org/trac/hackage/wiki/CabalInstall.

To install Turbinado:

git clone git@github.com:alsonkemp/turbinado-website.git cd turbinado-website cabal install [This might fail, saying that "trhsx" can't be found.  However, "trhsx" was built during the install and is probably at ~/.cabal/bin/trhsx, so copy "trhsx" to your path and re-run "cabal install".]

CGI Configuration

[Note: you don't want to use CGI without statically compiling in some Controllers, Layouts and Views.  See below.]

Usually, Turbinado is called with “-p” to specify the port the process should listen on (e.g. “turbinado -p 8080″).  When called with the “-c” flag, Turbinado will handle CGI requests.  However, because of the process setup and tear down times, responding to CGI requests takes about 250ms, which is considerably slower than responding to HTTP requests (about 1ms).

Again following Rails, Turbinado includes a CGI script called “dispatch.cgi” in the /static directory.

Apache Configuration

In order to use Turbinado’s CGI functionality with Apache, you’ll need to something like the following in order to tell Apache to allow CGI scripts in your Turbinado /static directory and to send all requests (e.g. “^(.*)$”) to the “dispatch.cgi” script.

  DocumentRoot /home/alson/turbinado-website/static
  <Directory "/home/alson/turbinado-website/static">
     Options +FollowSymLinks +ExecCGI +Includes
     AddHandler cgi-script .cgi
     AllowOverride None
     Order allow,deny
     Allow from all
  </Directory>
  RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.cgi [QSA,L]

Static Compilation Of Resources

Turbinado is designed to dynamically compile and link in the various resources (Controllers, Layouts, Views) needed to serve a request. However, it can take up to 15 seconds to complete that process the first time a particular page is requested (subsequent requests are very fast). With CGI, the server only ever sees the first request, so Turbinado would never be able to serve a CGI request faster than 10-20 seconds.

To fix this, you can now compile into the server particular resources. See Config/Routes.hs here. Turbinado stores a function along with the file path and function name in a tuple, so you just give Turbinado that information in the Routes.hs file and it’ll load those functions into the CodeStore at startup:

staticLayouts =
    [ ("App/Layouts/Default.hs", "markup", App.Layouts.Default.markup)
    ]

Support for “formats”

Rails has great support for file formats. Turbinado is trying to follow that lead. The system will try to figure out the MIME type based on the extension. According to the standard routes (Config/Routes.hs), the following path=>View mappings will occur:

/abba/ding => /App/Views/Abba/Ding.hs
/abba/ding.xml => /App/Views/Abba/DingXml.hs
/bloof/snort/1.csv => /App/Views/Bloof/SnortCsv.hs

The same Controller handles all formats; only the View will change. Also, usage of a format causes a blank Layout to be used (on the assumption that you don’t want a Layout used with a CSV, XML, etc output).

Lower case paths

Turbinado now defaults to using lower case paths (configured in Config/App.hs), so the following paths get mapped to Controllers and Views as follows:

/abba/ding_fling => /App/Controllers/Abba.hs : dingFling
                 => /App/Views/Abba/DingFling.hs

Cookies

Cookies are now supported. Examples here.

setCookie $ mkCookie "counter" (show 0)
v <- getCookieValue "counter"
deleteCookie "counter"

Cookie Sessions

Session data is encrypted and stuffed into a cookie. The encryption key is set in Config/App.hs. Session usage examples are here.

setSessionValue "counter" "0"
v <- setSessionValue "counter"
deleteSessionKey "counter"
abandonSession

Bodo Nauman (http://www.naumann.cc/) got Turbinado to build with GHC 6.10 (after forcing me to fix a bunch of recently added bugs…).  Details are here.  There’s currently an issue with the crypto Hackage package which doesn’t build with GHC 6.10**.  Fortunately, the darcs version of crypto builds fine.

Bodo beat me by a couple of days to an updated “How to install Turbinado” post.  Since v0.5 is freshly minted (ahem… and debugged) (and includes cookies, cookie sessions (lifted from Michael Snoyman’s hweb), a Rails-like respondTo function, …), I’ll post an update in a day or two.

Adolfo Builes also has started to replicate a Rails tutorial in Turbinado, so I hope I can point you to his results soon.

** The crypto build fails when building some test applications.  What’s weird is that “cabal install” installs the test applications into the bin directory.  Why would I want test applications in /usr/local/bin?  I don’t see a Cabal flag for “just build the libraries”…

From Happstack:

“[If HApps doesn't do anything...] the project will eventually be superceded by other Haskell frameworks like Turbinado which are not nearly as innovative”

Woohoo!  People are talking about Turbinado!

err… Wait! Turbinado is being dissed!

Turbinado vs. HApps

Each project is valid and valuable, so I’m hesitant to get to far into a X vs. Y discussion, but here are some aspects that I thought about when evaluating HApps and developing Turbinado:

• Turbinado is built on top of Nicklas Broberg’s HSP/HSPR, which I consider to be an innovative-port of ASP to Haskell…
• HApps is a much bigger project with many more features; Turbinado is small and needs your help [shameless plug: help here].
• HApps seems more like a library of really useful functions which is very flexible; Turbinado is more like a web app framework and provides a web server along with defined mechanisms for adding Controller, Views and Components.  Convention over Configuration = different styles, not better styles.
• HApps seems less than simple (see here); Turbinado is all about simple (see here)
• HApps has an aversion to relational databases; Turbinado observes that lots of people use relational databases and supports them.

Turbinado vs. HApps, again

To me, it seems as though the choice between HApps and Turbinado depends more on orientation than on functionality.  If you are interested in: opposing RDMBSs, but are into something like Sinatra, then HApps is probably the best choice; if you’re interested in writing web apps in the style of Ruby On Rails, then Turbinado (though young and pretty) may be the best choice, especially given the simplicity of building Turbinado (newly cabal installable!  to be described in a forthcoming post with tutorials, install details, singing, dancing, high kicks, etc).

Here’s To Success

Most successful languages have more than one web framework, so I hope for success for both HApps and for Turbinado. They’re very different frameworks and serve very different needs. The Haskell community would be well served if both frameworks survived and thrived.  As Merb and Rails have demonstrated, cross-pollination is excellent; the stronger HApps and Turbinado are, the more they can benefit each other.

Turbinado continues to evolve:

Turbinado (http://www.turbinado.org) is an easy 
to use Model-View-Controller-ish web framework for Haskell.


The source for the framework can be found at:
http://github.com/alsonkemp/turbinado

The source for the website turbinado.org can be found at:
http://github.com/alsonkemp/turbinado-website
(see the /App directory for the code for www.turbinado.org)

Release 0.4 contains:
* A dramatically improved ORM (or Type-Relation Mapper) 
   which handles foreign keys.  Still PostgreSQL only at this point.
* All dependencies in tmp/dependencies to ease building the application.
* In code documentation (not complete, but starting).
* Documentation (see http://turbinado.org/Architecture).

Release 0.5 will focus on:
* Ease of installation!
* Moving to GHC 6.10 (whenever Debian shifts).  Diego Eche 
   provided a port from plugins to ghc-api.
* Additional functionality (e.g. sessions, authentication, etc).
* Tutorials.

Updated per Ganesh’s comments.

Over here, Adolfo commented:

“Hi,I tried to do follow the example, but I couldn’t even install the packages, hsx and hs-plugins were impossible. I tried with cabal and manually, and neither of those worked . any suggestion, known issue with this packages?”

I’ve been busy adding features to Turbinado and haven’t circled back around to making sure that it’s easy to build, so I can claim a lot of the blame for the build problems.  Turns out to be really important that publicly released packages are easily buildable…

Thinking back, I have really struggled to build Turbinado… and I wrote Turbinado!  Turbinado depends on some particular bugfix-ish library releases (e.g. GSomething 0.6.1). With GHC 6.10, a bunch of libraries have broken or have changed so much that they badly break Turbinado. (I need to specify better the versions in turbinado.cabal.)

At times, I’ve considered bundling the dependencies into Turbinado so that building Turbinado would be easy, but that’s always felt like a cop-out. So I’m pleading for “cabal install”. Given Turbinado’s dependence on particular versions of libraries, I would love to able to do:

cabal install turbinado
  OR  (from /home/alson/turbinado)
cabal build

Cabal Install

Most casual users of Ruby, Python, Perl, Java, etc, know that those languages have automagic build/dependencies system (respectively, gem, eggs, CPAN, maven). The tools may be of varying quality, but many tutorials include something like “First, use GEM to install the package: gem install rails” and demonstrate just how simple it is to get a useful piece of software installed.

This is not the case in Haskell. I’d guess that no more than 5% of Haskellers know about the cabal command line tool and “cabal install”. On the other hand, I’d guess that 95% of novice Rubyers know about “gem install”. These automated build/dependency system are now critical to the success of languages. As a beginner in Ruby, I always knew that I could easily try out various libraries by using GEM to install bits of software. I’m now fairly experienced with Haskell and, partly because of that experience, I don’t believe that I can easily try out various Haskell libraries.

Niklas Broberg’s HSP is a great example of the challenge of building Haskell programs. HSP is very nicely separated into modular libraries which: makes it easy to apply pieces of HSP’s functionality to a program; makes it hard for a human (at least for me) to build any one part of HSP because each part depends on so many other parts. A build/dependency tool would make HSP much easier to build into existing programs.

The Plea

I love using Haskell and Haskell will only get better if more people are able to use it. IMO, a pre-condition to the growth of the language is a solid, easy to use build/dependency system. Cabal is that system for GHC and the cabal command line tool is a key part of that system.

Unfortunately, the cabal command line tool isn’t bundled with GHC, but … Please get it, build it, use it, report any bugs, compliment the Cabal team, etc. It’ll be a great help to the Haskell community.

darcs get http://darcs.haskell.org/cabal-install
cd cabal-install
sh bootstrap.sh

Update: Haskell Platform

Ganesh points out the Haskell Platform Proposal, so it looks as though there is a plan to incorporate the cabal command line tool. See the following:

http://www.haskell.org/haskellwiki/Haskell_Platform

http://www.haskell.org/haskellwiki/Haskell_Platform/FAQ

P.S. Anyone know if the cabal command line tool is going to make it into GHC?

Links to cabal install information:

http://hackage.haskell.org/trac/hackage/wiki/CabalInstall

http://hackage.haskell.org/cgi-bin/hackage-scripts/package/cabal-install

http://ghcmutterings.wordpress.com/2008/11/10/bootstrapping-cabal-install/

Updated to reflect modifications to the ORM (No “Model” suffix; fractured in Types, Functions, Relations)

These are very early days for Turbinado, so much change is going on… But here’s a quick tutorial on putting together a poor man’s page editor/manager in Turbinado.

Build Turbinado

Warning!: With 6.10’s changes in dynamic plugins, Turbinado only builds with GHC 6.8 right now. Fixing this is next up in the dev queue.

You’ll need to have the following packages installed to have a go at installation:

• GHC 6.8 (darcs)
• haskell-src-exts (darcs)
• harp (darcs)
• hslogger (git)
• encoding (darcs)
• hsx (darcs)
• hs-plugins (darcs)
• http (darcs)
• HDBC (git)
• HDBC-PostgreSQL (git)

Grab the code

git clone git://github.com/alsonkemp/turbinado.git

Build it

With all of the packages installed, wait for a new moon, stand on tip-toes, and do the following:

runghc Setup.lhs configure
runghc Setup.lhs build

Configure It

cp Config/App.hs.sample     Config/App.hs
cp Config/Routes.hs.sample Config/Routes.hs

Edit App.hs and Routes.hs to taste.

Create the Page table

CREATE TABLE page (
    title character varying NOT NULL,
    content text NOT NULL,
    _id character varying(255) NOT NULL PRIMARY KEY,
);

Updated: added PRIMARY KEY which is needed by the ORM generator.

Generate the ORM models

runghc scripts/GenerateModels

You should now have an interesting file or two in App/Models.  The files are organized as follows: * App/Models/Bases/Comment.hs – all base Models inherit this. * App/Models/Bases/AbcXyzType.hs – the representation of the database table abc_xyz. Don’t edit this! It’s autogenerated and your changes will get ignored on the next gen. * App/Models/Bases/AbcXyzFunctions.hs – CRUD functions on abc_xyz. Don’t edit this! It’s autogenerated and your changes will get ignored on the next gen. * App/Models/Bases/AbcXyzRelations.hs – functions on tables related to abc_xyz. Don’t edit this! It’s autogenerated and your changes will get ignored on the next gen. * App/Models/AbcXyz.hs – the user configurable area of the AbcXyz. By default this just imports the above three tables. All of your custom find, insert and update methods go here.

A big shout out to .netTiers for pointing the way on building a code-generator ORM.

Create your Layout

Just as in Rails, the Layout usually defines the majority of your sites page structure. See here for the Layout used for turbinado.org.

Create your Page controller

The controller handles the request and sets up ViewData for the View to render/display. This little Page controller has the following functions/methods/actions:

• index: list all Pages.
• show: render one Page.
• new: display a blank Page form.
• create: take the submission of the ‘new’ action.
• edit: display an existing Page in a form for editing.
• save: take the submission of the ‘edit’ action.

Full version here. Here’s a snippet:

-- 'id' is an important function name in Haskell, so I use id' for the Page's "id".  There's got
-- to be a better solution
 
-- This is the generated ORM for the 'page' table in the database.
import qualified App.Models.Page
 
-- Index lists out all of the pages
-- setViewDataValue is used to store data for retrieval by the View.  Idea lifted from ASP.NET 
-- http://msdn.microsoft.com/en-us/magazine/cc337884.aspx
index :: Controller ()
index  = do pages <- App.Models.Page.findAll --use ORM goodness to get all pages
            setViewDataValue "pages-list" $ map (\p -> (title p, _id p)) pages
 
-- Show shows one page
-- Notes:
--  * The "id" is parsed from the request's path and put into the Settings.
--     See: http://github.com/alsonkemp/turbinado-website/tree/master/Config/Routes.hs
--  * getSetting returns a "Maybe a".  getSetting_u is the unsafe version and returns just the "a".
show :: Controller ()
show  = do id'  <- getSetting_u "id"
           p <- App.Models.Page.find id'
           setViewDataValue "page-title" (title p)
           setViewDataValue "page-content" (content p)
 
-- snip --
 
save :: Controller ()
save = do   id'  <- getSetting_u "id"
            _title   <- getParam_u "title"
            _content <- getParam_u "content"
            p <- App.Models.Page.find id'
            App.Models.Page.update p {title = _title, content = _content}
            redirectTo $ "/Page/Show/" ++ id'

Create Your Views

This Controller requires 4 views:

• Index
• Show
• New
• Edit – this could probably be the same as New.

See here for pre-built views. Here’s the “Index” view (note: *this isn’t very sugary and that makes me sad *):

page =  <div>
          <h1>
            Page Index
          </h1>
          <% do ls <- getViewDataValue_u "pages-list" :: View [(String, String)]
                mapM indexItem ls
          %>
        </div>
 
-- fugly, but I'm still getting used to HSP-like templating
indexItem (t,i) = return $ cdata $ unlines $
                   ["<div style='padding: 0pt 5px;'>"
                   ," <a href=\"/Page/Show/" ++ i ++"\">"
                   ,"  "++ t
                   ," </a>"
                   ,"</div>"
                   ]

Here’s the “Show” view and it’s pretty sugary:

page = <div>
          <h1><% getViewDataValue_u "page-title" :: View String %></h1>
          <% getViewDataValue_u "page-content" :: View String %>
        </div>

Run It

Start up Turbinado:

dist/build/turbinado/turbinado -p 1111

Browse to it: http://localhost:1111/Page/Index (it’ll take a couple of seconds to compile the Model, Controller and View)

Examples!

There’s still lots of work to do on the ORM and on Documentation. As the code base matures, both should progress. Examples are a nice way to drive development forward and to engage the community, so let me know if you’d like to see anything in particular demonstrated and I’ll try to implement it.

For those of you interested in Turbinado, here’s a quick status update:

• I separated the code for the turbinado.org website from the code for the framework.  The framework is here and the website code is here.
• I’m going to finish up implementing HAML templating for Turbinado in the next few days.
• After HAML templates are in, I’ll provide a tutorial on implementing a mini-CMS/wiki in Turbinado (the code is already in the website.  The standard-Rails-ish “look, Mom!  No code!” type of tutorial.  Just enough to convince you to download it, but not enough to get you to be significantly productive. 
• Adam Stark is providing some greatly needed polish here as he attempts to get this beastie to build.  Turbinado really needs to be easier to build…
• Diego Echeverri is doing some work to get Turbinado to work with GHC 6.10 here. I had a difficult time getting my HSP-ish View templates working with 6.10, so I hope Diego can do it. I’d greatly prefer to be working with 6.10, but I couldn’t get there…

Writing a little web framework turns out to be a lot of work (it’s all the little stuff (documentation!!) that really gets ya).  I’ve greatly appreciated the ability to build on the work of others (especially Niklas Broberg, Don Stewart, Bjorn Bringert and John Goerzen) and am grateful that others are providing help to this fledgling project.