A whirlwind tour

I've been on a bit of a tour of late.

I've tried out JetBrain's MPS system, attempting to use it for code generation for work, and I couldn't get it to do what I needed it to. In a lot of ways, the experience reminded me of attempting to use ANTLR's AST treegen approach - I was frustrated by a lack of debugging support.

I am having more success with using ANTLR to parse an input file, in this case a Java interface, to my own Java AST classes, and then generating code using StringTemplate. My first bash at this approach mushed the input AST and the output AST into one set of classes, and that was kinda messy. Next time I'm going to separate the concerns of the input AST and the output AST required by StringTemplate. I'm thinking I'll attempt to build the output AST as an adapter over the input AST, instead of having a AST modification pass. I'll find out how that goes in the coming week.

I'm also playing with Terracotta, the network attached memory store for the JVM. The one downside of Terracotta is that it exposes something that is often hidden - Java's classloaders. In java, a classes' identity not only includes it's bytecode, but also which classloader it was loaded by. As Terracotta is all about storing java objects while maintaining their identity, this means that classloaders become important. Now, given I am attempting to marshall classes between tomcat instances an Mule instances, I am having to subvert the classloaders of both. Not fun. Not fun at all.

The one thing that is becoming stupidly obvious about now is that the future of computing is about scaling sideways. Terracotta is an attempt to bring this to the Java world, but it feels like a bolt on. It breaks so many assumptions that Java programmers have. It makes thread programming with locks important, and honestly lock based concurrency is a bitch to get right.

One approach is to do Clojure + Terracotta. Another is to go back to Erlang. Interesting times.

Generating code...

Like pretty much every programmer I know, I'm trying to automate what I do. What this amounts to, is that I'm trying to generate code to eliminate the boilerplate from my life.

For a variety of reasons, I get paid to code in Java. One of the most verbose languages on the planet. I'd like to be able to rip out code generators to generate all the boiler plate i need to keep the kids from hurting themselves on the sharp knives.

I've tried ANTLR, but honestly it's tree transformation language is in need of better tooling support to help with debugging. I'm part way through trying to fathom JetBrain's MPS. It's complicated as all hell, but on the upside, it understands the Java AST. It's at about this point that I contemplate whether I should just bite the bullet and do it the traditional way, with lisp macros all the way.

The upside of using MPS is that it knows Java, and it comes with a decent editing environment. The downside of MPS is that it has all the complexities of building a editor, instead of just using emacs like everyone else. I don't think i'm ready to give up on MPS yet. I like the pretty.

Brain melt

I'm trying to implement a webapp api in Clojure inspired by the elegant model of Seaside. There are two important shifts in this port, firstly migrating from SmallTalk's object model to lisp's macro based approach, and secondly taking advantage of Clojure's concurrency primitives.

My brain is gently melting out my ears as a result. On the upside, when I get this to work, I get a surprisingly easy way to implement co-operative web applications. A wonderful environment for trying out new approaches to user interface design. It's just the game of getting from vision to implementation.

Sometime next year, at this rate. =)

So where now?

I have a question, but first I need to draw out some data points.

Firstly, we have Eucalyptus, a project that aims to replicate AWS's Elastic Computing Cloud APIs. This is being used by Sun and others to pitch data centers on the idea of being able to compete in the Cloud Computing game. In short, cloud computing is about to get comoditised.

Secondly, Erlang, Haskell, Scala, Clojure and various other languages are chasing how to best expose concurrency and network IO in a programmer friendly way. I believe we are on the cusp of beating this challenge. Actually, I think Clojure already has, but it's going to take a while for the reality to shake out. And for programmers to either accept they have to learn a new language, or go and open a cafe.

Given the two points above, the game change I see is that it now becomes pizza + beer pricing to host a webapp on a EC2 node that can service 2,000+ concurrent long poll connections. Development could be done on a $20/month node, early roll out on $50/month node, and when it gets really popular you can scale up to the 15.5G of ram beast for $800 a month. But all those prices are about to fall massively, thanks to the return of Moore's law, and the explosion of AWS EC2 wannabes.

So, what can you do with a server capable of mainting say 5K connections? Think FaceBook, but real time. Think livejournal meets twitter. Think corporate business workflow applications with the ability to co-operatively (aka, socially), work with co-workers.

In short, we are about to see a qualitative shift in the way people work together. And this will enable people to act together in ways that will make the 5k person parties organised over fb look positively pale in comparison. Yes, this is going to threaten pretty much everyone in a position of power, be that government, business, or religion. It's about to become affordable for everyone to host real time communication hubs.

On the upside, this means that we are about to start a process of learning how to group communications well. Honestly, everything we've done so far has sucked, and sucked badly. The biggest thing I'm getting out of Gladwell's Outliers is that up until now we have badly wasted the one reasource this planet that can make a difference. Us.

Rebuilding the world. Again.

I'm a little intimidated by the power of Clojure. It changes the dynamics of designing software. Traditionally it was important to get the overall design right because once coding started, changing the design suddenly became quite expensive. Changing the number of fields on a form has rippling changes right through the code base.

Not so in the world of properly factored Clojure development, because all the source code dependent on the shape of the forms is generated using macros. This changes design time from before coding to afterwards. This makes the design iterative, and done upon the working code.

I can see why lisp wizards were both prized and feared. They can build systems that cannot be built other ways, but at the same time it takes Gladwell's 10,000 hours to be ready for Lisp. As a lisp developer you need to understand the impacts of design changes right through the system, from the user interface to disk layouts, because the macros you write cover the whole solution space.

So I finally have an answer to why Lisp was never popular. It's not popular because it is genuinely hard to develop in. The repl is intimidating the same way a drag car is. Sure, it is a beast of enormous power, but it also has no time for formalities. If you don't know what you are doing, this beast will show you up, and probably hurt you in the process.

On the upside, I am now seriously looking at commercial software that costs thousands of dollars per year, per seat, and feeling that I can build it in a reasonably short time frame, once I have written a bunch of macros. Mmmm. Nothing like a challenge to get me going =)

It's knowledge, not intelligence.

First off, an admission. I'm probably the dumbest of my clan. I have next to no artistic ability, i'm not musically inclined, and I don't have the razor sharp wit. In short, I'm the runt of the litter.

In some ways, knowing I'm not the smartest has been an advantage. I've kept at things long after my other siblings gave up. I'm nowhere near Gladwell's 10,000 hours for mastery of any craft, but I've kept at computers long enough to be able to get things done.

It's why I'm sad about my brother. He's as smart, if not smarter, than I am. Yet he has wasted his talents by being combative with people who were trying to help him. He was so busy trying to prove his smarts that he never had time enough to pick up the knowledge.

In short, his insecurity driven rampages damaged the very friendships upon which his success was reliant.

My head hurts.

I've just spent three days trying to understand Clojure's for macro. And I reckon I have another couple of days worth of work pulling it apart yet.

It's a bit of a disservice calling it for, as it is actually a list comprehension, but it is called for because this is the construct to use when a traditional programmer would reach for a for loop.

It has iteration, variable binding, and choice. It is actually a baby programming language in a 30 odd line macro. I can see the parallels between this macro and the de-sugaring that haskell does with the monadic do blocks.

But still, it is a mind warping exercise pulling apart this macro. Fun tho. =)

Convergence

Clojure, a Lisp for the JVM with STM and Actors. Bringing the power of macros and functional programming to concurrency.

Terracotta, a network attached memory for clustering JVMs and persisting objects across JVM restarts.

jQuery, a JavaScript native api for manipulating the DOM using CSS3 selectors.

Raphael, a JavaScript library for creating vector graphics on the web, using SVG on Firefox, WebKit and Opera, and VML on Internet Explorer.

GMail outage...

A couple of days ago there was a GMail outage. Short summary, a combination of taking down a data center and a bug in the load distribution code leads to overloaded data center, and thus a bunch of people can't get to their mail.

A friend ribbed me about the readyness of Cloud Computing, on the back of this outage. My initial reaction was to defend Google on uptime stats alone. Now that Google have spread the love and admitted the cause, I'm going out on a limb and stating that very few (if any) corporate IT shops are prepared for dealing with entire data centers being taken off line for maintenance.

I mean, sure, there are corporates who have back up data centers, and even have machines in stand by DR mode. I'd be exceedingly surprised, however, if everything actually worked. Most DR tests I've seen were epic fails, and the fun thing is that DR readyness tests usually happen once a year, if at all.

If a corporate wants DR to work, it has to be a regular thing. Schedule DR cut over every friday night, and keep repeating until it works flawlessly. And then bring it back every month to make sure it keeps working flawlessly. Otherwise the business as usual is going to be in for a shock when things go pear shaped for real.

System integration nightmares

I'm currently poking my way through an enterprise service bus, figuring out how to make it do stuff. I've just successfully written my most complicated hello world program to date. It includes message formatters, message routers, two seperate processing pipelines, and it's processed across a SOAP interface generated from a generated WSDL file. Over engineering for the win, right?

The underlying metaphor embedded in the configuration language is one of a pipeline. Pipe line stages can have filters on both input and output, the output stage can select the next stage depending on attributes of the message, and the center of the stage, the component, does some form of transformation, all the while being isolated from the message transformation and routing happening around it.

The issue I'm having with this metaphor is that pipe lines are not turing complete. I need the ability to take a message in over a SOAP connection, do a bunch of steps (including writing to our local database as well as querying and updating a pair of remote ERP systems), and then return a result over that said same SOAP connection.

Now, in the simple examples, this return can be done, because an input endpoint can have metaphor breaking property of being inout as opposed to just in. What I'm scrabbling through the documentation attempting to understand is how i pass this SOAP connection through the chain so that I can return the result, even after a long trail of transforms. Not sure if it's going to work, or if I'm just going to have to cheat...

The most interesting point to me is that the pipeline metaphor is insufficient.

Thoughts...

Dad told me a story, a long time ago. It goes something like this.

Ted, a manager is asked by the CEO to come to dinner to discuss the possibility of stepping up to divisional chief. So on the appointed day, Ted clocks off after a day of fighting fires, goes home, gets changed, and heads in for dinner with the CEO.

On the way in, there is a terrible accident in a tunnel, and Ted is stuck in instant gridlock. (This story dates from the days before mobile phones, go with me on this one.) So after fighting his way through the accident debris, he arrives at his dinner appointment only an hour late. After explaining why he was late, due to the tragic circumstances in the tunnel, Ted was promptly told by the CEO that he wouldn't be getting the Divisional Chief position.

At the time I thought the CEO was being overly harsh on Ted, given that the circumstances surrounding Ted being late were outside of Ted's control. It is only now, looking back, that I understand the CEO's point of view.

I'm increasingly seeing that people split into one of two pretty neatly divided categories. Those who do things, and those whom things happen to. The language used by the two groups are largely not interoperable, let alone expectations over behaviour.

Odd, but true.

Code organisation...

There's a Law floating around somewhere about how compilers will have as many passes as the organisation that is creating it has groups. In short, the construction of the software is based on the communication fault lines of it's creator.

Traditional web apps have similar break downs, there are the sysadmins who wear metal shirts and have long hair, the database admins who are uptight and wear business shirts and demand respect for their models, the webapp tier guys who wear business shirts badly, and the web devs who wear jeans and indie tshirts, and frequently don't make it in on mondays.

These groups hate each other with a passion, and it shows in the application.

With the upcoming change of pushing the database out into the web client, and the back end becoming nothing but a persistent change notification hub, all the skills these current groups have in isolation need to be all brought to bear in the front end client.

By now I'm resigned to the fact that we are going to lose a generation of developers over this. We have on each other major change or technology. It's funny, but it took talking about all this at length with Dad to realise that we have seen all this before, and we will see it again.

A generated future

One of the things I like about Lisp is the fact that having a powerful macro capability in the language allows for concision in the code base that isn't available any other way. Unfortunately this leads to programmers minds exploding - powerful macros take time and patience to grok.

Near as I can tell, the other option for handling complexity also leads to programmer minds to explode. Instead of having an in language macro capability, I'm winding up generating code. The generated code, given it's weight in LoC has a similar mind bending capacity as Lisp macros, but the generated code is what is checked into revision control.

Is there a way out of this?