Hackfoofery

Alson Kemp

Archive for the ‘Turbinado’ Category

Organizing Terraform Projects

without comments

At Teckst, we use Terraform for all configuration and management of infrastructure.  The tidal boundary at the intersection of infrastructure and application configuration is largely determined by which kinds of applications will be deployed on which kinds of infrastructure.  Standing up a bunch of customized EC2 instances (e.g.Cassandra)?  Probably something that Ansible or Chef is better suited to as they’re built to step into a running instance and create and/or update its configuration (though, certainly, this can be done via Terraform and EC2 User Data scripts).

Teckst uses 100% AWS services and 100% Lambda for compute so we have a much more limited need.  We need Lambda Functions, API Gateways, SQS Queues, S3 Buckets, IAM Users, etc to be created and wired together; thereafter, our Lambda Function are uploaded by our CI system and run over the configured AWS resources.  In this case, Terraform is perfect for us as it walks our infrastructure right up to the line at which our Lambda Functions take over.

Terraform’s documentation provides little in the way of guidance on structuring larger Terraform projects.  The docs do talk about modules and outputs, but no fleshed-out examples are provided for how you should structure your project.  That said, many guides are available on the web ([1][2][3] are the top three Google results as of this writing).

Terraform Modules

Terraform Modules allow you to create modules of infrastructure which accept/require specific Variables and yield specific Outputs.  Besides being a great way to utilize third-party scripts (e.g. a script you find on Github to build a fully configured EC2 instance with Nginx fronting a Django application), Modules allow a clean, logical separation between environments (e.g. Production and Staging).  A good example of organizing a project using Terraform Modules is given in this blog post.  Initially, we approached organizing scripts similarly:

/prod/main.tf
/prod/vpc.tf - production configs for VPC module 
/staging/main.tf
/staging/vpc.tf - staging configs for VPC module
/modules/vpc/main.tf - contains staging configs for VPC module
/modules/vpc/variables.tf
/modules/vpc/outputs.tf

Now all of our prod configuration values are separate from our staging configuration values.  The prod and staging scripts could reference our generic vpc Module.  Initially, this seemed like a huge win.  Follow on to find out how it might not be a win for in-house-defined infrastructure.

Terraform Modules and First-Party Scripts

While Terraform Modules are a great way to use third party Terraform scripts to build your infrastructure, they felt awkward when using only first party scripts/definitions.  Reviewing other ways of organizing the scripts, they seemed to rely on Copy-and-Paste and that’s a Bad Thing, right?  But what if Copy-and-Paste is unavoidable?  With Terraform Modules, each module defines a set of Variables it accepts and it defines a list of Outputs it outputs.  In order to utilize Modules in first-party circumstances, you will need to determine a naming scheme for the Variables and Outputs (when using a third-party module, you must adopt their naming scheme).  Any mismatches between the names of Variables or Outputs will yield baffling error messages.  So a lot of time is spent threading Variables up into sub-Modules only to grab and thread Outputs from those sub-Modules down and then up into other sub-Modules via other Variables.  The best example of this is VPC information: nearly every Module will need to have the VPC.id threaded up into the module.  The safest way to make sure your Variable declarations line up with the Module’s Variable requirements is to Copy-and-Paste the names to-from the Modules.  Consider your VPC and Route53 scripts:

/main.tf
/terraform.tfvars
/vpc/main.tf
/vpc/outputs.tf
/vpc/variables.tf
/route53/main.tf
/route53/outputs.tf
/route53/variables.tf

We’ll need to:

  1. In main.tf: reference the vpc Module, passing in appropriate Variables.
  2. In /vpc/variables.tf: define Variables this Module will accept.
  3. In /vpc/outputs.tf: map the Module’s configuration values to the Module’s Outputs.
  4. In /vpc/main.tf: create the VPC using the Variables specified in /main/variables.tf.
  5. In /main.tf: reference the route53 Module, passing in as Variables the Outputs retrieved from the vpc Module.
  6. Then do Steps 2-5 for the /route53 Module.

The above has three significant issues:

  1. It’s tedious.  That’s something like 11 steps to create a VPC and Route53 domain.  That’s about 8 lines in a single Terraform script…  Here we’ve got 2-3 times that scattered across 7 files.
  2. It’s error-prone.  Didn’t get your naming convention right for one of the Outputs of some Module?  That’s going to happen.  And the Terraform errors aren’t too helpful in finding the issue.
  3. It’s based on Copy-and-Paste!  The root of all evil!  Sure, you don’t need to Copy-and-Paste but you’re going to do so as, for the hundredth time, you write into a Variable declaration the name of a Module’s Output…  And as you mispell the Output‘s name…

Less-Bad Terraform Organization

So I’m not sure we can avoid Copy-and-Paste.  If we can’t avoid it, can we organize our Terraform scripts in such a way that we minimize likely harm?  We approached it as follows, attempting to:

  • Strictly limit the surface area of the production and staging configurations so that it was obvious which was which.
  • Eliminate the differences in everything outside of the above differences in production and staging configurations areas.
  • Make clear how to migrate staging infrastructure alterations forward to production infrastructure.
  • Make clear where Copy-and-Paste is allowed.
  • Simplify our usage of Terraform.

We wound up with the following (very simplified) organization:

/prod/vpc.tf
/prod/route53.tf
/prod/variables.tf
/staging/vpc.tf
/staging/route53.tf
/staging/variables.tf

That’s it.  What’s this get us:

  • Very simple Terraform usage.  No need to trace Variables and Outputs up/down through Modules.  route53.tf can directly reference aws_vpc.main.id.
  • Very limited surface area for configuration.  All environment specific configuration values go in variables.tf.  It’s easy to compare the prod and staging variables.tf to see new or changed variables.
  • Simple migration to prod.  Just meld (or diff) staging and production.  We know the variables.tf will differ, so, with suitable review, the content of staging can be quickly and easily pulled into production.

An example of the content of /staging/variables.tf:

########################
# ACCOUNT SETTINGS
########################
variable "environment"    {default = "staging"}
variable "account_id"     {default = "acct-staging"}
variable "account_number" {default = 123456789 }
variable "aws_region"     {default = "us-east-1"}

########################
# DOMAIN SETTINGS
########################
variable "root_domain"         {default = "not-real.com"}
variable "root_private_domain" {default = "not-real.internal"}

Additional Considerations/Practices

Is it that simple?  Not quite, but it’s close.  We have some development resources in our staging environment that don’t exist in our production environment (looking at you crazy MS SQL Server instance) but it’s pretty obvious to anyone involved in the infrastructure that those resources are staging only.

Currently, one or two people do all of the infrastructure development, so we haven’t really wrestled with conflicts in the terraform.tfstate but that will be an issue regardless of the organization of Terraform files.

Written by alson

February 14th, 2018 at 11:26 am

Posted in Turbinado

Parsing a DICOM file for fun and profit

without comments

For various reasons, I needed to have a CT scan of my jaw done.  This is a very high resolution 3D scan.  The technician needed to review the scan to make sure it was of high quality and I stood behind him and looked over his shoulder.  The software was pretty impressive, but the 3D model and resolution were really impressive.  And then I left the office and drove home…

… and as I was driving, I thought: wouldn’t it be fun to have a copy of the data?; perhaps I could build a point cloud and shoot it into a crystal (as I’d done with fractals)?  So I called back the lab (Park XRay) and asked if I could have a copy of the data.  “Sure!  It’s your skull.” was the reply and they delivered an extra copy to my dentist.

The files were in DICOM format and were produced or for-use by iCATVision.  Fortunately, Python has a DICOM library, so it was fairly easy to parse the files.  My code is on GitHub.  [The code is not pretty, but it worked.]

I’ve previously “printed” point clouds into crystals using Precision Laser Art, so I needed to convert the 448 16-bit slices of my jaw into 1-bit XYZ point clouds.  “visualize.py” provides a simple 2D visualization of the slices.  Most importantly, it let me tune the threshold values for the quantizer so that the point cloud would highlight interesting structures in my jaw.  Here’s the interface (perhaps it’s obvious, but I’m not a UX expert…):

Once I’d tuned the parameters, I added those parameters to “process.py” and generated the giant XYZ point cloud.  The format of the point cloud is just:

X1 Y1 Z1

X2 Y2 Z2

X3 Y3 Z3

[repeat about 3 million times...]

I sent my order to Precision Laser Art and, after 7 days and $100, received this:

Which had a nicely cushioned interior:

And this is the resulting crystal.  It’s the C-888 80mm cube.

While it’s not amazingly easy to see in this photo, my vertebrae and hyoid bone are clearly visible in the crystal.

Anyhow, the point is: medical data is cool.  You can get it, so get it and play with it!  😉

Written by alson

January 18th, 2013 at 11:34 pm

Posted in Turbinado

Django: handling configs for multiple environments

without comments

A common way to manage settings across environments in Django is to create a local_settings.py file and then copy into it environment specific settings during deployment.  Although much of our web work is done in Django now, the Rails way of managing environments is superior.

In your project, create a settings_env directory and put into it local.py, dev.py, etc files for environment specific setup.

## ^^ standard settings.py above
# Import environment specific settings
# Pull in the settings for specific environments
# It's the last argument.
env = os.environ.get('DJANGO_ENV')

if env == "production" : from settings_env.dw      import *
elif env == "staging"  : from settings_env.staging import *
elif env == "dev"      : from settings_env.dev     import *
elif env == "local"    : from settings_env.local   import *
else: 
    print "######################################################"
    print " No environment specified or specified environment"
    print " does not exist in /settings_env/."
    print " Continuing with no settings overrides."
    print " To specify an environment (e.g. production), use"
    print "  DJANGO_ENV=production ./manage.py runserver"
    print "######################################################"
    quit()
if DEBUG=True:
  ## ^^ settings.py for DEBUG = True below

Written by alson

October 7th, 2012 at 2:42 pm

Posted in Turbinado

Tunnel MySQL over SSH to remote server issue

without comments

There are a million pages about this, but I just bumped into a tricky issue and figured I’d drop a quick note about it.

First off, tunnel MySQL to the server (on a UNIXy box) by doing:

ssh  -L 33306:localhost:3306 your-server.com

Simply, that tells SSH to listen on local port 33306, a port chosen because it’s an obvious variation on MySQL’s default of 3306).  When something connects to that port, SSH will accept the connection and will forward it to the remote host, which will then connect it to the appropriate host and port on the far side.  In this case, we’re asking the server to connect to port 3306 on localhost, but you could connect to any server and any port.

The tricky issue was that on my Debian laptop, MySQL uses socket file for communcation even if you specify a port.  So this will fail and have you talking with your local MySQL instance:

mysql --port=33306  -u your_remote_user -pyour_remote_password

In order to force MySQL to use TCP (instead of sockets), you can force the connection protocol or specify a host (note: you can’t use ‘localhost’; you need to use 127.0.01):

mysql --protocol=tcp --port=33306 \
   -u your_remote_user -pyour_remote_password
mysql -h 127.0.0.1 --port=33306  \
   -u your_remote_user -pyour_remote_password

Written by alson

September 7th, 2012 at 2:29 pm

Posted in Turbinado

Getting high on your own stash…

without comments

Lots of talk around right now about how distracted we are by technology. Too much reporting-speak about it, though. Here’s a different, more personal take:
http://tweetagewasteland.com/2010/06/say-hello-to-my-little-friend/

Great quote:

When the WiFi went down during the official iPhone 4 demo, didn’t you sort of wish Steve Jobs would turn to the crowd and say, “You know what, let’s just talk.”

Written by alson

June 18th, 2010 at 5:24 pm

Posted in Turbinado

How do you maintain type-safety in web applications?

with 2 comments

Note: this post really concerns statically typed languages, but my little bits of example code are in Ruby since that’s the MVC implementation I know best.

I like MVC-ish-ness for web applications

The MVC-ish aspect of Rails is much clearer than the ASP/PHP-ish world’s chaos, so I’m bought into MVC-ish as a better way.  Separating the request processing into MVC generally gets me into the right mindset when I’m looking at individual bits of code. That said, Rails’ MVC still falls to satisfy fully my cravings for a web framework.

I don’t like the lack of type-safety

As Curt and I were discussing over in my last post, I am bothered by the lack of type-safety in MVC web frameworks.  In most MVC frameworks, type-safety is ignored or broken, e.g.:

  • Rails is basically not typed and Views can access the Controller’s instance variables;
  • ASP.NET uses an object called ViewState to pass data from Controllers to Views, but you have to cast ViewState to the right type in the View [yes, I’m treating ASP.NET as MVC…];
  • Turbinado copied ASP.NET’s ViewState and inherits its issues.

I’d like to figure out a way to maintain type-safety within my web application.  In the case of dynamic languages (e.g. Ruby), I’d like “type-clarity” so that I’m _nearly_ guaranteed of having the right type in the right place. For example [using Ruby syntax]:

The header of a web app might have a account section which behaves differently whether or not a User has been loaded into @user in the Controller.   It’s easy enough to have the section behave differently based on whether @user.nil?, but I can’t guarantee that, if the @user is not nil, @user is of type User.  *I* can certainly check the class of @user, but that leaves me open to errors at runtime when I _do_ change the type of @user and forget to update all of my type checks.

I want to be able to set the @user variable to a User in the Controller and have that type information enforced at compile time in the View.  Given an MVC-ish system, there seems to be two ways to do this:

  • Use different data types for each view and piece of a view.   This would be a nightmare since you’d have to define hundreds of separate types, build them differently and then send the right type to the right view.
  • Only ever create the values in question right near their usage, but this pushes Model fiddling and request params access down into the View, so doesn’t fit the web MVC style…

A Slight Modification to Rails’ MVC-ish-ness

Even if it isn’t the Rails MVC Way, it seems as though a reasonable way to maintain type safety would be to:

  • Don’t use Controller to set up variables for use in Views.
  • Mix more of the Controller functionality into the Models so that Models can manage creation of themselves.
  • Have Views use Models directly (and have Models cache/memo-ize themselves as necessary).
  • Use Controllers *only* for validation and routing.
  • Have Views play directly with the Models they require.

Currently, Rails’ MVC works something like the following:

---
some_controller.rb
class SomeController < ApplicationController
  def index
    @user = User.find(params[:user_id])
  end
end
---
some_view.html.erb
 
    <= HEY! Is @user **really** a User?
 
  Log in!
 
---

Instead, how about giving the Model access to the params and letting it use its business rules to set itself up:

---
some_controller.rb
class SomeController < ApplicationController
  def index
  end
end
---
some_view.html.erb
 
       <= HEY! That's almost guaranteed to work.
 
  Log in!
 
---
user_model.rb
Class User < ActiveRecord::Base
  def self.from_request
    find(params[:user_id])  <= the models is accessing the Environment...
  end
end

This is wrong, right?

Per Terrence Parr in Enforcing Strict Model-View Separation in Template Engines [PDF]:

[…] there is widespread agreement that keeping the view as simple and free from entanglements with model and controller as possible is a worthwhile ideal to strive for.

[…] Strictly separating model and view is an extremely desirable goal because it increases flexibility, reduces maintenance costs, and allows coders and designers to work in parallel.

But I think Parr’s point is really that business logic shouldn’t appear in Views not that UserModel access shouldn’t be done in a View.  Or is there some horrible breakdown that would occur if Models had access to request parameters and Controllers didn’t set up Models?

An issue that does arise with this modification is increased testing complexity when the Environment is accessible to Models.  Whereas now Models take a few parameters and then pass or don’t pass the test, allowing Models to access the Environment would mean that tests would have to set up request parameters, cookies, etc.  Simple, little, lovely Unit tests suddenly get larger and potentially uglier…

Type-safety + MVC ~= peanut butter & chocolate

Is type-safety a worthy goal to try to attain in MVC-ish web frameworks?  If so, how is the best way of doing so? In Rails, judicious application of filters and good naming schemes can get you “type-confident”, but you’re never type-safe.  Perhaps Rails has just the right blend for an MVC-ish web framework…

If MVC-ish web frameworks aren’t the way to go, what organization is better? Templating systems that only allow template markup/commands seem pretty safe, but I’d seriously miss the simplicity of mixed markup/code. ASP.NET compiles the template and then the “code behind” gets well-typed access to the XML, but I really prefer MVC to that organization.

Written by alson

March 18th, 2010 at 8:02 pm

Posted in Turbinado

Ruby vs. JRuby: URI.parse fails with percent signs

with 7 comments

Ran into a nasty issue with URL parsing a while ago in a Rails app.  I was pulling URLs in from Amazon’s shopping APIs and started receiving occasional InvalidURIError exceptions.  Tracked it down to percent signs in the URLs passed from Amazon.  Percent signs are kinda reserved for escaping/encoding some characters in URLs, but, while I’m not sure that the Amazon URLs are truly legal, the parsing function shouldn’t fail on a non-standard usages.  After all, the web is a messy place and failing-safe on parsing messy data seems a better behavior…    Examples:

irb(main):001:0> require 'uri'
=> true
irb(main):002:0> u = URI.parse("http://www.yahoo.com/test?string=thing%ding") # questionable
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/test?string=thing%ding
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in `split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:in `parse'
 from (irb):3

irb(main):003:0> u = URI.parse("http://www.yahoo.com/abba?test%20text")  # legal usage
=> #<URI::HTTP:0x164cbde URL:http://www.yahoo.com/abba?test%20text>

irb(main):004:0> u = URI.parse("http://www.yahoo.com/abba?test%u2020text") # quasi-legal usage
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/abba?test%u2020text
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in `split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:in `parse'
 from (irb):5

Enter JRuby

Been thinking about checking out JRuby since I would then have access to the universe of Java libraries.  So Ruby’s URI.parse method is not very robust and I should have caught the exception, tried an alternate parser, etc, but what if I used the Java URL decoder on these URLs?  Works as expected:

irb(main):001:0> include_class "java.net.URLDecoder"
=> ["java.net.URLDecoder"]

irb(main):014:0> u = URL.new("http://www.yahoo.com/test?string=thing%ding")                       
=> #<Java::JavaNet::URL:0x13e4a5a>                                                                
irb(main):015:0> u.getQuery                                                                       
=> "string=thing%ding"                            

irb(main):013:0> u = URL.new("http://www.yahoo.com/test%u2020tring")
=> #<Java::JavaNet::URL:0x3aef16>
irb(main):014:0> u.path
=> "/test%u2020tring"

Lesson

Ruby’s a great little language; Java’s a great big library => JRuby’s a great little language with a great big library.

Also, an interesting article on URL parsing from DaringFireball.

Written by alson

November 27th, 2009 at 4:21 pm

Posted in Turbinado

Announce: Turbinado V0.7

without comments

Turbinado has been making solid progress.  See the GitHub repo.

Added a couple of big new features:

HAML and XHTML Templates

While Turbinado used XHTML for templates before, the HSX library (an amazing piece of work) was a real pain to use and threw very strange errors. 

HSX (and its associated libraries) has been torn out of Turbinado and has been replaced by a custom preprocessor called ‘trturbinado’. The preprocessor is run by the webserver over any templates (or views) before they’re compiled and loaded. The preprocessor handles XHTML and a limited version of HAML (to be enhanced).

See the example below. After preprocessing, the ‘someHtml’ and ‘someHAML’ functions will yield the exact sasme code as the ‘someTextXHTML’ function.

Note: I’m not great at writing parsers, so I wouldn’t doubt that there are bugs in the preprocessor!

markup :: VHtml
markup=
  <div>
    <% someTextXHtml %>
    <% someHtml %>
    <% someHAML %>
  </div>
 
-- | These are the raw Turbinado.View.Html style tags
someTextXHtml :: VHtml
someTextXHtml = do s <- getViewDataValue_u "sample_value" :: View String
                   ((tag "div") (
                    ((tag "i") (stringToVHtml s)) +++
                    ((tag "b") (stringToVHtml "some text in Turbinado.View.Html style"))
                    ))
 
someHtml :: VHtml
someHtml = do s <- getViewDataValue_u "sample_value" :: View String
              <div>
                <i><%= s %></i>
                <b>some text in XHtml style</b>
              </div>
 
someHAML :: VHtml
someHAML = do s <- getViewDataValue_u "sample_value" :: View String
              %div
                %i= s
                %b some text in HAML style

FastCGI Support

FastCGI has been an often requested feature, so I rewrote the CGI handling and added FastCGI support, too. Much as with CGI, using FastCGI is very simple. Just tell Apache to use ‘dispatch.fcgi’ to handle requests.

<VirtualHost *>
  DocumentRoot /home/alson/projects/turbinado/static
  <Directory "/home/alson/projects/turbinado/static">
    Options +FollowSymLinks +ExecCGI +Includes
    AddHandler cgi-script cgi
    AddHandler fastcgi-script fcgi
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>
  RewriteEngine On
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f
  RewriteRule (.*) $1 [L]
  # Use FCGI
  RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]
  # Use CGI
  # RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]
</VirtualHost>

Performance of Turbinado’s FastCGI interface is pretty good, but is well short of Turbinado’s HTTP interface, so, if you can live without the benefits of FastCGI, HTTP, which is simpler and faster, is the way to go.

Written by alson

June 14th, 2009 at 12:15 am

Posted in Turbinado

Turbinado V0.6.5

with 3 comments

Been a long silent few months (blog cliche, anyone?), but Turbinado is cranking along again. Bodo (http://www.naumann.cc/) is using Turbinado for a school project so we’ve been working together to advance Turbinado. So what’s new? Mostly touch ups:

HTTP 4k Support

The HTTP 3000 package seems to be slowing down while HTTP 4000 seems to be taking over.  Makes sense to update Turbinado to the latest version.  Done.

Better Forms Support

Added in support for multipart forms, so now you can upload files to Turbinado.  Note: there is currently no limit on the size of the upload, though.  (See Turbinado.Environment.Files for how to access uploaded files.)

Improved ORM

The PostgreSQL ORM is pretty sweet and now it’s even better.  Improved error handling, quoting of tables names, etc.  Next step is to build the MySQL ORM.

Summary

We’re moving again.  Note: turbinado-website is currently updated, but turbinado isn’t (it’ll be fixed by June 4th).

Would love your feedback on what could be done to improve Turbinado.  Although I made a point of running turbinado.org on Turbinado, it may be time to switch to a CMS that allows people to contribute to the project more easily.

Written by alson

June 3rd, 2009 at 2:24 am

Posted in Haskell,Turbinado

ANNOUNCE: Turbinado V0.6

with 14 comments

It’s been long enough since Turbinado V0.4 that I figured I’d skip V0.5 and go straight to announcing Turbinado V0.6.  Lots of new excellent features:

  • By popular demand, support for CGI serving. Apparently some web hosts don’t support HTTP proxying, so some folks requested CGI support.
  • Statically compiled Layouts, Views, Controllers.
  • Support for “.format” in routes. If a request path is “/User/List.xml”, then the following View will be called: /App/Views/User/ListXml.hs.
  • Lower case paths.
  • Support for cookies (see here for examples).
  • Encrypted cookie sessions (see here to see how to use them).
  • Much easier installs using cabal-install.
  • Support for GHC 6.10.  GHC 6.8 is no longer supported.

Turbinado V0.7 will be all about:

  • Documentation. (seriously.)
  • User authentication.
  • Tutorials.

Installation

Installation is pretty painless if you use cabal-install, so make sure that you have cabal-install installed first.  See here: http://hackage.haskell.org/trac/hackage/wiki/CabalInstall.

To install Turbinado:

git clone git@github.com:alsonkemp/turbinado-website.git
cd turbinado-website
cabal install
[This might fail, saying that "trhsx" can't be found.  However, "trhsx" was built during the install and is probably at ~/.cabal/bin/trhsx, so copy "trhsx" to your path and re-run "cabal install".]

CGI Configuration

[Note: you don’t want to use CGI without statically compiling in some Controllers, Layouts and Views.  See below.]

Usually, Turbinado is called with “-p” to specify the port the process should listen on (e.g. “turbinado -p 8080”).  When called with the “-c” flag, Turbinado will handle CGI requests.  However, because of the process setup and tear down times, responding to CGI requests takes about 250ms, which is considerably slower than responding to HTTP requests (about 1ms).

Again following Rails, Turbinado includes a CGI script called “dispatch.cgi” in the /static directory.

Apache Configuration

In order to use Turbinado’s CGI functionality with Apache, you’ll need to something like the following in order to tell Apache to allow CGI scripts in your Turbinado /static directory and to send all requests (e.g. “^(.\*)$”) to the “dispatch.cgi” script.

  DocumentRoot /home/alson/turbinado-website/static
  &lt;Directory "/home/alson/turbinado-website/static"&gt;
     Options +FollowSymLinks +ExecCGI +Includes
     AddHandler cgi-script .cgi
     AllowOverride None
     Order allow,deny
     Allow from all
  &lt;/Directory&gt;
  RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.cgi [QSA,L]

Static Compilation Of Resources

Turbinado is designed to dynamically compile and link in the various resources (Controllers, Layouts, Views) needed to serve a request. However, it can take up to 15 seconds to complete that process the first time a particular page is requested (subsequent requests are very fast). With CGI, the server only ever sees the first request, so Turbinado would never be able to serve a CGI request faster than 10-20 seconds.

To fix this, you can now compile into the server particular resources. See Config/Routes.hs here. Turbinado stores a function along with the file path and function name in a tuple, so you just give Turbinado that information in the Routes.hs file and it’ll load those functions into the CodeStore at startup:

staticLayouts =
    [ ("App/Layouts/Default.hs", "markup", App.Layouts.Default.markup)
    ]

Support for “formats”

Rails has great support for file formats. Turbinado is trying to follow that lead. The system will try to figure out the MIME type based on the extension. According to the standard routes (Config/Routes.hs), the following path=>View mappings will occur:

/abba/ding => /App/Views/Abba/Ding.hs
/abba/ding.xml => /App/Views/Abba/DingXml.hs
/bloof/snort/1.csv => /App/Views/Bloof/SnortCsv.hs

The same Controller handles all formats; only the View will change. Also, usage of a format causes a blank Layout to be used (on the assumption that you don’t want a Layout used with a CSV, XML, etc output).

Lower case paths

Turbinado now defaults to using lower case paths (configured in Config/App.hs), so the following paths get mapped to Controllers and Views as follows:

/abba/ding_fling => /App/Controllers/Abba.hs : dingFling
                 => /App/Views/Abba/DingFling.hs

Cookies

Cookies are now supported. Examples here.

setCookie $ mkCookie "counter" (show 0)
v <- getCookieValue "counter"
deleteCookie "counter"

Cookie Sessions

Session data is encrypted and stuffed into a cookie. The encryption key is set in Config/App.hs. Session usage examples are here.

setSessionValue "counter" "0"
v <- setSessionValue "counter"
deleteSessionKey "counter"
abandonSession

Written by alson

March 12th, 2009 at 10:20 pm

Posted in Haskell,Turbinado