Alson Kemp

My Apache process is only using one core!

Written by alson

January 4th, 2012 at 11:41 pm

Posted in Geekery

Bookmark and Share

I was recently working on a client site (a good-sized one) and was checking on the health of their application servers.  I noticed that each of their app servers was running a few of the cores much harder than the other cores.  This was in the evening and they get most of their traffic during the day; it runs Django under mod_wsgi in daemon mode with 8 processes and 25 threads per process.  Further, the boxes were not VPSs/VMs, but were dedicated, multicore boxes.  So they had multicore hardware and the web server was running in a multicore friendly way.

At the time, the load for each box was around 0.5.  And various process IDs rotated as the top CPU users, so process IDs weren’t bound to a core.  The ratio of traffic between the cores (ignoring actual core number and focusing on the utilization of each core, since each box was different) was something like:

Core # : Core Utilization

1      : 15%

2      : 2%

3      : 1%

*      : 0%

So why would one processor bear most of the load?  I googled and googled, and found little useful information.   I banged on one server with Apache’s benching tool (“ab”) while watching core utilization and, sure enough, all cores shared the load equally.  So what was going on?

I’m not sure if it’s the Linux kernel or a natural outcome of CPU caches, but the simplest explanation is that in low load situations processes are similar, due to cache coherence, will flock to the same core.  Rather than spreading a set of processes across a set of cores that don’t necessarily share the same cache, processes naturally gravitate to the cores that experience the lowest cache misses.

Upshot: it’s rational for the system to schedule most operations of a process or group of similar processes on one core when a system is relatively lightly loaded.  This is especially true if the cores are “Hyperthreading” and are sharing resources (read: their caches)!

with one comment

ExtJS 4 Models + node.js

Written by alson

February 22nd, 2011 at 2:50 pm

Posted in Geekery

Bookmark and Share

Finally starting to play with node.js.   Also, getting back into developing with the lovely ExtJS.  ExtJS 4 added strong support for client side models.

I was thinking that it’d be nice to share a lot of code for models between the client and server.   Turns out that it’s not that difficult.   Super quick-n-dirty code below.  Now the question is: how much duplication can be removed from client and server models? I don’t want to include all server-side code in client-side code, so might do something like:

  • /common/models/user.js – ExtJS model;
  • /client/models/user.js – tune the model for client side (e.g. add a REST connection to the server);
  • /server/models/user.js – includes client/models/user.js;  overrides critical bits (e.g. the Proxy); adds a bunch of server specific code.

If all of my models are in model.*, then I can probably iterate through them and auto-generate Mongoose models when the server boots…  Fun.

This is definitely a hack, but isn’t as frighteningly ugly as I expected:

fs = require('fs');

// stub out a fake browser for ext-core-debug.js
navigator = {};
window = {
  navigator : 'Linux',
  attachEvent: function() {return false;}
navigator = {'userAgent' : 'node'};
document = {
  getElementsByTagName : function () {return false;}};

// Helper function 
function injectJS(f) {
  eval(fs.readFileSync(f, encoding="ascii"));

//Pull in ExtJS components

// Register the model
Ext.regModel('models.User', {
      fields: [
          {name: 'name',  type: 'string'},
          {name: 'age',   type: 'int'},
          {name: 'phone', type: 'string'},
          {name: 'alive', type: 'boolean', defaultValue: true}
    validations: [
        {type: 'presence',  field: 'age'},
        {type: 'length',    field: 'name',     min: 2}
    changeName: function() {
        var oldName = this.get('name'),
        newName = oldName + " The Barbarian";
        this.set('name', newName);

// Create an instance
var user = Ext.ModelMgr.create({
      name : 'Conan',
        age  : 24,
        phone: '555-555-5555'
}, 'models.User');

// Use the instance
user.get('name'); //returns "Conan The Barbarian"
user.fireEvent('changed', 'my hair');

repl = require("repl");
repl.start('ExtJS> ');

with 9 comments

KickLabs (SF Incubator)

Written by alson

July 16th, 2010 at 2:18 pm

Posted in Geekery

Bookmark and Share

Great incubator just opened in downtown San Francisco: KickLabs.  Ridiculously great space, a great team and a list of exciting events.  Definitely a place to get to know.

And they welcome entrepreneurs of all ages!

without comments

Getting high on your own stash…

Written by alson

June 18th, 2010 at 5:24 pm

Posted in Turbinado

Bookmark and Share

Lots of talk around right now about how distracted we are by technology. Too much reporting-speak about it, though. Here’s a different, more personal take:

Great quote:

When the WiFi went down during the official iPhone 4 demo, didn’t you sort of wish Steve Jobs would turn to the crowd and say, “You know what, let’s just talk.”

without comments

NXDom rocks…

Written by alson

March 29th, 2010 at 1:48 pm

Posted in Tools

Bookmark and Share

A couple of months ago, Johann Rocholl announced NXDom.  Basically, Johann created an algorithm to build millions of likely-to-be-interesting possible domain names and then queried DNS servers to see if the servers knew of the possible domain names.  Any domains that got NX (non-existent domain) responses from DNS servers were entered into the NXDom database as available.  Johann added a nice query interface and, voila!, changed how I search for domains.

Search for Domains V1

I don’t have any particular recipe for searching for domains, but I do something like:

  1. Think about my target market.
  2. Think up a few keywords.
  3. Try the keywords at http://www.instantdomainsearch.com.
  4. Find most are registered.
  5. Think up other less interesting keywords…
  6. Find one kinda crappy domain name that isn’t registered and register it because I’m frustrated…
  7. Dwell on the fact that I had a crappy domain name…

Search for Domains V2 (New and Improved with NXDom)

  1. Think about my target market.
  2. Think up a few keywords.
  3. Try the keywords at NXDom.
  4. See 300 results for each keyword.
  5. See lots of possibilities for great domains for each keyword.
  6. Register a great domain (making sure to click the NXDom affiliate link so that Johann gets his rev share).
  7. Dwell on my super cool domain.

The best part of NXDom is that it works very well for very popular terms.  A friend is in the CFO business, so we searched on domains ending in “CFO” and found numerous interesting domains:


I don’t know Johann and I have no affiliation with NXDom.  Everyone to whom I show NXDom gets wide-eyed and starts thinking up domain ideas, so I figured that I’d share more broadly. NXDom has certainly changed, and greatly improved, the way I search for domains.

with one comment

How do you maintain type-safety in web applications?

Written by alson

March 18th, 2010 at 8:02 pm

Posted in Turbinado

Bookmark and Share

Note: this post really concerns statically typed languages, but my little bits of example code are in Ruby since that’s the MVC implementation I know best.

I like MVC-ish-ness for web applications

The MVC-ish aspect of Rails is much clearer than the ASP/PHP-ish world’s chaos, so I’m bought into MVC-ish as a better way.  Separating the request processing into MVC generally gets me into the right mindset when I’m looking at individual bits of code. That said, Rails’ MVC still falls to satisfy fully my cravings for a web framework.

I don’t like the lack of type-safety

As Curt and I were discussing over in my last post, I am bothered by the lack of type-safety in MVC web frameworks.  In most MVC frameworks, type-safety is ignored or broken, e.g.:

  • Rails is basically not typed and Views can access the Controller’s instance variables;
  • ASP.NET uses an object called ViewState to pass data from Controllers to Views, but you have to cast ViewState to the right type in the View [yes, I’m treating ASP.NET as MVC…];
  • Turbinado copied ASP.NET’s ViewState and inherits its issues.

I’d like to figure out a way to maintain type-safety within my web application.  In the case of dynamic languages (e.g. Ruby), I’d like “type-clarity” so that I’m _nearly_ guaranteed of having the right type in the right place. For example [using Ruby syntax]:

The header of a web app might have a account section which behaves differently whether or not a User has been loaded into @user in the Controller.   It’s easy enough to have the section behave differently based on whether @user.nil?, but I can’t guarantee that, if the @user is not nil, @user is of type User.  *I* can certainly check the class of @user, but that leaves me open to errors at runtime when I _do_ change the type of @user and forget to update all of my type checks.

I want to be able to set the @user variable to a User in the Controller and have that type information enforced at compile time in the View.  Given an MVC-ish system, there seems to be two ways to do this:

  • Use different data types for each view and piece of a view.   This would be a nightmare since you’d have to define hundreds of separate types, build them differently and then send the right type to the right view.
  • Only ever create the values in question right near their usage, but this pushes Model fiddling and request params access down into the View, so doesn’t fit the web MVC style…

A Slight Modification to Rails’ MVC-ish-ness

Even if it isn’t the Rails MVC Way, it seems as though a reasonable way to maintain type safety would be to:

  • Don’t use Controller to set up variables for use in Views.
  • Mix more of the Controller functionality into the Models so that Models can manage creation of themselves.
  • Have Views use Models directly (and have Models cache/memo-ize themselves as necessary).
  • Use Controllers *only* for validation and routing.
  • Have Views play directly with the Models they require.

Currently, Rails’ MVC works something like the following:

class SomeController < ApplicationController
  def index
    @user = User.find(params[:user_id])

    <= HEY! Is @user **really** a User?

  Log in!


Instead, how about giving the Model access to the params and letting it use its business rules to set itself up:
class SomeController < ApplicationController
  def index

       <= HEY! That's almost guaranteed to work.

  Log in!

Class User < ActiveRecord::Base
  def self.from_request
    find(params[:user_id])  <= the models is accessing the Environment...

This is wrong, right?

Per Terrence Parr in Enforcing Strict Model-View Separation in Template Engines [PDF]:

[…] there is widespread agreement that keeping the view as simple and free from entanglements with model and controller as possible is a worthwhile ideal to strive for.

[…] Strictly separating model and view is an extremely desirable goal because it increases flexibility, reduces maintenance costs, and allows coders and designers to work in parallel.

But I think Parr’s point is really that business logic shouldn’t appear in Views not that UserModel access shouldn’t be done in a View.  Or is there some horrible breakdown that would occur if Models had access to request parameters and Controllers didn’t set up Models?

An issue that does arise with this modification is increased testing complexity when the Environment is accessible to Models.  Whereas now Models take a few parameters and then pass or don’t pass the test, allowing Models to access the Environment would mean that tests would have to set up request parameters, cookies, etc.  Simple, little, lovely Unit tests suddenly get larger and potentially uglier…

Type-safety + MVC ~= peanut butter & chocolate

Is type-safety a worthy goal to try to attain in MVC-ish web frameworks?  If so, how is the best way of doing so? In Rails, judicious application of filters and good naming schemes can get you “type-confident”, but you’re never type-safe.  Perhaps Rails has just the right blend for an MVC-ish web framework…

If MVC-ish web frameworks aren’t the way to go, what organization is better? Templating systems that only allow template markup/commands seem pretty safe, but I’d seriously miss the simplicity of mixed markup/code. ASP.NET compiles the template and then the “code behind” gets well-typed access to the XML, but I really prefer MVC to that organization.

with 2 comments

Reflections on leaving Haskell

Written by alson

March 13th, 2010 at 5:02 am

Posted in Haskell

Bookmark and Share

Update: Dammit but that’s a great set of comments…  Haskellers rock…  I wrote this post fairly quickly and didn’t expect it to get quite the attention it got, so please read the comments.  There’s a lot of good stuff in them.

Update #2: a comment over on Hacker News points out that the reasons I give are shallow…  I agree.  It works out that simple, shallow issues can become big issues over time and as a system grows larger and larger.  That was exactly the reason I started with Haskell: to wear the hair shirt.  As my systems got larger, the shirt just got too itchy.

I picked up Haskell during MIT’s IAP ’04 (during B-school…) figuring that I should learning a new language during the winter.  I’m an experienced coder, didn’t need yet another object oriented programming language and couldn’t get OCaml to compile my “hello world” program.  Haskell was the first challenging computer language I’d seen in a while and I was quickly addicted (though not quickly productive).

As might be obvious, I haven’t been able to spend much time on Turbinado over the past year.  I have been able to spend a bit of time thinking about Haskell and I think that I’ve left the Haskell camp…  Here’s why.

Things I Could Do Without

No subclassing of datatypes

In Turbinado I passed an Environment back through the function hierarchy and I wound up building the Environment out of a bunch of Maybe types that I would incrementally fill in.

data Environment = Environment {
           getCodeStore :: Maybe CodeStore,
           getRequest  :: Maybe (HTTP.Request String),
           getResponse :: Maybe (HTTP.Response String) }

Yuck…  What I really wanted to do was build a number of EnvironmentXs so that I could safely use EnvironmentX or EnvironmentY in a particular function.  But I would have to define two separate data types in order to avoid the Maybes:
data Environment1 = Environment {
           getCodeStore :: CodeStore,
           getRequest :: HTTP.Request String }

data Environment2 = Environment {
           getCodeStore ::  CodeStore,
           getRequest :: HTTP.Request String,
           getRequest :: HTTP.Request String }

What I really wanted to do is say that Environment2 is Environment1 with a bit more stuff, e.g.:
data Environment1 = Environment {
           getCodeStore :: CodeStore,
           getRequest :: HTTP.Request String }

data Environment2 = Environment1 with {
           getResponse :: HTTP.Response String }

(This has type system implications that I’m sure are more complicated than I’d understand…)

(I’m sure that there’s a monad that could solve this problem, but see below…)

Dependence on Monads is not a Good Thing

Monads are a brilliant idea… on paper.  I’ll be the first to say that I think that my level of intelligence hampers my ability to really work with monads, but I’m a reasonably intelligent guy so I suspect that a lot of coders share this complaint.  Lots of other functional-ish languages have added monad libraries, but I think that they’ve done so more because monads are useful for DSLs than because monads are a foundational building block for the language.

In the real world, monads seem quite constraining.  Composing them is rough and the composed monads are difficult to understand.  I’m ashamed to say that one of the first things I did in Turbinado was to tear down the complicated monads that I’d copied from HSP.

The IO Monad makes me unhappy

It’s something that’s been sitting in the back of my mind and I finally let it bubble up. I feel kinda dirty when I use the IO Monad, as though I’m cheating on the language. I understand that functional programs have to do IO and that the IO Monad is an elegant solution… but I’m still uncomfortable. The majority of the functions in Turbinado are in the IO Monad… and that bothers me.  Half of my code looks like the following and that makes me unhappy because it isn’t functional-ish:

simpleFunction s = do i <- otherFunction s
                      putStrLn i

Why don’t I just use a language that’s friendly to IO? My question exactly.

Things I Loved

Functional Programming

I’m sold. Seriously. That’s fantastic stuff right there. And without Haskell I would be trapped in Scala-land, writing syntax-sugared-Java, confused about why functional programming was so great…


Haskell is fantastic because it works as advertised.  I considered learning other functional, but was struck by their functional impurity.  Looking back on Haskell, I love the design of the language.  It’s an architectural masterpiece and you gotta love that (though obviously I’ve decided not to live in it…).

Wearing the Hair Shirt

My hat is off to the Haskell community.  SPJ and Wadler did the programming world a service by creating Haskell.  Wearing the Hair is hard and it’s very worthwhile.  It’s why I decided to learn Haskell and it made me a better, smarter developer.

The Community

Who doesn’t like working with people who’re smarter than they are?


There’s probably more I love about Haskell and more that I could do without, but I think that I’m moving on to Clojure.   Rich Hickey has a strong love for functional programming coupled with a good sense of the compromises that can be made for the language to be very productive.  The JVM is great: it runs my Rails apps, is speedy, does great garbage collection, etc.  I wish it had a dialect of ML or Haskell on it, but I’m going to try to teach myself to look past the parentheses and get back into LISP.  It’s been 22 years since I typed a line of LISP, but this dog can learn new tricks… especially with the benefit of the last 5 mind-bending years with Haskell.

Truth be told I’m a little sad.  I’ve kinda felt part of a secret society with Haskell and it’s a bummer to be leaving the secret Haskell society…

with 44 comments

HighCharts, Bluff, gRaphael – Javascript charting libraries

Written by alson

December 2nd, 2009 at 8:49 pm

Posted in Geekery,Tools

Bookmark and Share

Bumped into HighCharts today.  See here: http://highcharts.com Very nice library for embedding charts into your HTML.  Nice to see that browsers are finally getting to the point that HighCharts can be written.   Also, it’s nice to think about being able to dump Flash charting solutions in favor of a simple Javascript library.  Worth the $80 for commercial use (I don’t know about you, but I’ve spent well more than $80 of my time wrestling with an open source Flash charting library).


[Updating post to add more libs.]

Okay.  I got called out for not poking around a bit more.  There are nice open-source, liberally licensed JS chart libs.

Open Source

Bluff : Ruby’s Gruff ported to JS.  Looks pretty straightforward.

gRaphael : built on the awesome Raphael JS vector graphics library.  Limited functionality, but that’ll change.

Canvas3D : super cool, but not sure if it’s ready for a production site…

jqPlot : charting for jQuery.  I’m neither pro- nor con-jQuery, so I’m not sure that I’d pick a jQuery-centric library.

PlotKit : dependent on MochiKit.

Flot : charting for jQuery.  Again, since I use different JS libraries in different circumstances/apps, I’m not excited about having to pull in jQuery, but Flot looks nice.

Commercial License

JSCharts : HighCharts looks prettier…  $129 for a license for all of your sites (vs. $360 for HighCharts)

Emprise Charts : Very nice.  Pricing is inline with JSCharts and HighCharts.


with 9 comments

Ruby vs. JRuby: URI.parse fails with percent signs

Written by alson

November 27th, 2009 at 4:21 pm

Posted in Turbinado

Bookmark and Share

Ran into a nasty issue with URL parsing a while ago in a Rails app.  I was pulling URLs in from Amazon’s shopping APIs and started receiving occasional InvalidURIError exceptions.  Tracked it down to percent signs in the URLs passed from Amazon.  Percent signs are kinda reserved for escaping/encoding some characters in URLs, but, while I’m not sure that the Amazon URLs are truly legal, the parsing function shouldn’t fail on a non-standard usages.  After all, the web is a messy place and failing-safe on parsing messy data seems a better behavior…    Examples:

irb(main):001:0> require 'uri'
=> true
irb(main):002:0> u = URI.parse("http://www.yahoo.com/test?string=thing%ding") # questionable
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/test?string=thing%ding
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in `split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:in `parse'
 from (irb):3

irb(main):003:0> u = URI.parse("http://www.yahoo.com/abba?test%20text")  # legal usage
=> #<URI::HTTP:0x164cbde URL:http://www.yahoo.com/abba?test%20text>

irb(main):004:0> u = URI.parse("http://www.yahoo.com/abba?test%u2020text") # quasi-legal usage
URI::InvalidURIError: bad URI(is not URI?): http://www.yahoo.com/abba?test%u2020text
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:436:in `split'
 from /home/alson/bin/../apps/jruby-1.4.0/lib/ruby/1.8/uri/common.rb:484:in `parse'
 from (irb):5

Enter JRuby

Been thinking about checking out JRuby since I would then have access to the universe of Java libraries.  So Ruby’s URI.parse method is not very robust and I should have caught the exception, tried an alternate parser, etc, but what if I used the Java URL decoder on these URLs?  Works as expected:

irb(main):001:0> include_class "java.net.URLDecoder"
=> ["java.net.URLDecoder"]

irb(main):014:0> u = URL.new("http://www.yahoo.com/test?string=thing%ding")                       
=> #<Java::JavaNet::URL:0x13e4a5a>                                                                
irb(main):015:0> u.getQuery                                                                       
=> "string=thing%ding"                            

irb(main):013:0> u = URL.new("http://www.yahoo.com/test%u2020tring")
=> #<Java::JavaNet::URL:0x3aef16>
irb(main):014:0> u.path
=> "/test%u2020tring"


Ruby’s a great little language; Java’s a great big library => JRuby’s a great little language with a great big library.

Also, an interesting article on URL parsing from DaringFireball.

with 7 comments

Announce: Turbinado V0.7

Written by alson

June 14th, 2009 at 12:15 am

Posted in Turbinado

Bookmark and Share

Turbinado has been making solid progress.  See the GitHub repo.

Added a couple of big new features:

HAML and XHTML Templates

While Turbinado used XHTML for templates before, the HSX library (an amazing piece of work) was a real pain to use and threw very strange errors. 

HSX (and its associated libraries) has been torn out of Turbinado and has been replaced by a custom preprocessor called ‘trturbinado’. The preprocessor is run by the webserver over any templates (or views) before they’re compiled and loaded. The preprocessor handles XHTML and a limited version of HAML (to be enhanced).

See the example below. After preprocessing, the ‘someHtml’ and ‘someHAML’ functions will yield the exact sasme code as the ‘someTextXHTML’ function.

Note: I’m not great at writing parsers, so I wouldn’t doubt that there are bugs in the preprocessor!

markup :: VHtml
    <% someTextXHtml %>
    <% someHtml %>
    <% someHAML %>
-- | These are the raw Turbinado.View.Html style tags
someTextXHtml :: VHtml
someTextXHtml = do s <- getViewDataValue_u "sample_value" :: View String
                   ((tag "div") (
                    ((tag "i") (stringToVHtml s)) +++
                    ((tag "b") (stringToVHtml "some text in Turbinado.View.Html style"))
someHtml :: VHtml
someHtml = do s <- getViewDataValue_u "sample_value" :: View String
                <i><%= s %></i>
                <b>some text in XHtml style</b>
someHAML :: VHtml
someHAML = do s <- getViewDataValue_u "sample_value" :: View String
                %i= s
                %b some text in HAML style

FastCGI Support

FastCGI has been an often requested feature, so I rewrote the CGI handling and added FastCGI support, too. Much as with CGI, using FastCGI is very simple. Just tell Apache to use ‘dispatch.fcgi’ to handle requests.

<VirtualHost *>
  DocumentRoot /home/alson/projects/turbinado/static
  <Directory "/home/alson/projects/turbinado/static">
    Options +FollowSymLinks +ExecCGI +Includes
    AddHandler cgi-script cgi
    AddHandler fastcgi-script fcgi
    AllowOverride None
    Order allow,deny
    Allow from all
  RewriteEngine On
  RewriteRule (.*) $1 [L]
  # Use FCGI
  RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]
  # Use CGI
  # RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/dispatch.fcgi [QSA,L]

Performance of Turbinado’s FastCGI interface is pretty good, but is well short of Turbinado’s HTTP interface, so, if you can live without the benefits of FastCGI, HTTP, which is simpler and faster, is the way to go.

without comments