Testing with Kittens

This is a test post. It is written in markdown in an editor and published to Blogger. I'm curious how it is publishing.

And now a photo of two kittens, from flickr.com.

Written with StackEdit.

Ranting quietly...

One of the odd things about working here is that we are constantly reminded that we can change the world. Thus the serious lack of rants since I joined the 'plex. It is both humbling and empowering to know that changing the world is not only a possibility, but it is a duty. So, here I am, quietly changing the world. It's fun. You should do it too =)

Testing

Let's see if Blogger API posts get automatically shared to Google+ ...

From winter to summer in just fifteen hours...

Oh, hello San Fran, what a lovely summer you have...

And for something completely different, I'm talking at the Sydney Hackathon, at Sydney Uni. Come along, it's bound to be a lot of fun!

Sydney Startup Weekend this weekend!

This weekend I will be one of the mentors at the Sydney Startup Weekend. It starts tonight at 6:30pm at the Australian Technology Park, in Everleigh. Come along, it's going to be a great weekend!

Not quite so ranty Mouse

I got asked at a user group last week why I've stopped ranting. I spent a good ten years ranting across a wide variety of blogging platforms, and since becoming a DevRel at Google, I've stopped.

I had to think for a couple of days as to why I haven't felt the need to get it off my chest of late. The short summary is that joining Google has highlighted the fact that all the tools you need to build world beating software are already out there. It's just a matter of investing the time in learning the tools, the technology, and mastering a domain that interests you.

Interested in mobile phones? Learn Android. Interested in web applications? AppEngine and GWT give you enormous power. Want to build games? WebGL is up and running in the latest Chrome and Firefox betas, with Opera not too far behind. And if you are really hardcore, Native Client is shaping up as a good way to ship C++ code to clients without having to deal with cross platform and logitics head aches.

I always thought that the geniuses at Google had access to amazing tools that allowed them to build applications like Google Docs that were beyond the reach of mere mortals. Now that I have spent time amongst Googlers, I've realised that all of these applications were built by very smart, very dedicated teams. That level of dedication is open to anyone, if they choose to chase their dreams...

Chinatown

Testing, testing, 1... 2... 3...

How can you tell the difference between a drummer and a roadie? How do you know if the stage is level? How can you tell it's been too long since I've been on stage?

I start making drummer jokes. Oh, and testing blogger / buzz integration. There is a reason for this post, but it hasn't happened yet. You could say this post is a stepping stone to a bigger post, a better future... and more drummer jokes.

It's wet outside, but the music is cool

A thought experiment

I remember back in the day when companies needed to raise millions of dollars to set up their web farm. One of the upsides of doing that was that people thought big, they dreamed of taking over the world. These days of running a clustered multi-tier web app stack on AWS for pocket change has lead to people having small dreams.

So, how about we do some maths. Let's say you have $10k a day in budget to burn on compute resources. What could you do with it? Let's take numbers from AWS, they quote Linux Double Extra Large High Memory instances at $1 an hour. So, at 24 hours a day, we have about 416 instances. That is approximately 13TB of RAM, and over 1.6k cores. Now what can you do with that much RAM and CPU?

Cool preso

A very cool analysis of a website's copy. Worthy of a detailed read.

Recommendations for bettermeans

View more presentations from Shereef Bishay.

Mmm, stuff.

It's been almost a year since I last posted here. In the mean time I've wandered a bit, I spent a bunch of time playing with Google Wave, toying with Google Buzz, and poking at the corpse of LiveJournal. What brought me back? The fact that blogger has a new template editing system that doesn't suck nearly as badly as it did. In fact, it's kinda decent. About the only thing i want in it is a "round the corners and add subtle shadows" button. And maybe a Web2.0 Always in Beta sticker.

In some ways, this feels like i've been full circle. I'm back to writing Antlr grammars. Antlr 3.2's tree filters are wickedly cool, and Terence Parr's Language Implementation Patterns is thoroughly hurting my noodle. It's a damn fine read, if you enjoy LL(*) and syntactic predicates.

One of the things I am playing with at the moment is stuffing Antlr generated parsers inside of web browsers using Google's GWT. It opens a wild vista of new potentials, of being able to edit and hack seriously complicated models inside a browser. Revenge of the nerds, web2.0 style.

A whirlwind tour

I've been on a bit of a tour of late.

I've tried out JetBrain's MPS system, attempting to use it for code generation for work, and I couldn't get it to do what I needed it to. In a lot of ways, the experience reminded me of attempting to use ANTLR's AST treegen approach - I was frustrated by a lack of debugging support.

I am having more success with using ANTLR to parse an input file, in this case a Java interface, to my own Java AST classes, and then generating code using StringTemplate. My first bash at this approach mushed the input AST and the output AST into one set of classes, and that was kinda messy. Next time I'm going to separate the concerns of the input AST and the output AST required by StringTemplate. I'm thinking I'll attempt to build the output AST as an adapter over the input AST, instead of having a AST modification pass. I'll find out how that goes in the coming week.

I'm also playing with Terracotta, the network attached memory store for the JVM. The one downside of Terracotta is that it exposes something that is often hidden - Java's classloaders. In java, a classes' identity not only includes it's bytecode, but also which classloader it was loaded by. As Terracotta is all about storing java objects while maintaining their identity, this means that classloaders become important. Now, given I am attempting to marshall classes between tomcat instances an Mule instances, I am having to subvert the classloaders of both. Not fun. Not fun at all.

The one thing that is becoming stupidly obvious about now is that the future of computing is about scaling sideways. Terracotta is an attempt to bring this to the Java world, but it feels like a bolt on. It breaks so many assumptions that Java programmers have. It makes thread programming with locks important, and honestly lock based concurrency is a bitch to get right.

One approach is to do Clojure + Terracotta. Another is to go back to Erlang. Interesting times.

Generating code...

Like pretty much every programmer I know, I'm trying to automate what I do. What this amounts to, is that I'm trying to generate code to eliminate the boilerplate from my life.

For a variety of reasons, I get paid to code in Java. One of the most verbose languages on the planet. I'd like to be able to rip out code generators to generate all the boiler plate i need to keep the kids from hurting themselves on the sharp knives.

I've tried ANTLR, but honestly it's tree transformation language is in need of better tooling support to help with debugging. I'm part way through trying to fathom JetBrain's MPS. It's complicated as all hell, but on the upside, it understands the Java AST. It's at about this point that I contemplate whether I should just bite the bullet and do it the traditional way, with lisp macros all the way.

The upside of using MPS is that it knows Java, and it comes with a decent editing environment. The downside of MPS is that it has all the complexities of building a editor, instead of just using emacs like everyone else. I don't think i'm ready to give up on MPS yet. I like the pretty.

Brain melt

I'm trying to implement a webapp api in Clojure inspired by the elegant model of Seaside. There are two important shifts in this port, firstly migrating from SmallTalk's object model to lisp's macro based approach, and secondly taking advantage of Clojure's concurrency primitives.

My brain is gently melting out my ears as a result. On the upside, when I get this to work, I get a surprisingly easy way to implement co-operative web applications. A wonderful environment for trying out new approaches to user interface design. It's just the game of getting from vision to implementation.

Sometime next year, at this rate. =)

So where now?

I have a question, but first I need to draw out some data points.

Firstly, we have Eucalyptus, a project that aims to replicate AWS's Elastic Computing Cloud APIs. This is being used by Sun and others to pitch data centers on the idea of being able to compete in the Cloud Computing game. In short, cloud computing is about to get comoditised.

Secondly, Erlang, Haskell, Scala, Clojure and various other languages are chasing how to best expose concurrency and network IO in a programmer friendly way. I believe we are on the cusp of beating this challenge. Actually, I think Clojure already has, but it's going to take a while for the reality to shake out. And for programmers to either accept they have to learn a new language, or go and open a cafe.

Given the two points above, the game change I see is that it now becomes pizza + beer pricing to host a webapp on a EC2 node that can service 2,000+ concurrent long poll connections. Development could be done on a $20/month node, early roll out on $50/month node, and when it gets really popular you can scale up to the 15.5G of ram beast for $800 a month. But all those prices are about to fall massively, thanks to the return of Moore's law, and the explosion of AWS EC2 wannabes.

So, what can you do with a server capable of mainting say 5K connections? Think FaceBook, but real time. Think livejournal meets twitter. Think corporate business workflow applications with the ability to co-operatively (aka, socially), work with co-workers.

In short, we are about to see a qualitative shift in the way people work together. And this will enable people to act together in ways that will make the 5k person parties organised over fb look positively pale in comparison. Yes, this is going to threaten pretty much everyone in a position of power, be that government, business, or religion. It's about to become affordable for everyone to host real time communication hubs.

On the upside, this means that we are about to start a process of learning how to group communications well. Honestly, everything we've done so far has sucked, and sucked badly. The biggest thing I'm getting out of Gladwell's Outliers is that up until now we have badly wasted the one reasource this planet that can make a difference. Us.

Rebuilding the world. Again.

I'm a little intimidated by the power of Clojure. It changes the dynamics of designing software. Traditionally it was important to get the overall design right because once coding started, changing the design suddenly became quite expensive. Changing the number of fields on a form has rippling changes right through the code base.

Not so in the world of properly factored Clojure development, because all the source code dependent on the shape of the forms is generated using macros. This changes design time from before coding to afterwards. This makes the design iterative, and done upon the working code.

I can see why lisp wizards were both prized and feared. They can build systems that cannot be built other ways, but at the same time it takes Gladwell's 10,000 hours to be ready for Lisp. As a lisp developer you need to understand the impacts of design changes right through the system, from the user interface to disk layouts, because the macros you write cover the whole solution space.

So I finally have an answer to why Lisp was never popular. It's not popular because it is genuinely hard to develop in. The repl is intimidating the same way a drag car is. Sure, it is a beast of enormous power, but it also has no time for formalities. If you don't know what you are doing, this beast will show you up, and probably hurt you in the process.

On the upside, I am now seriously looking at commercial software that costs thousands of dollars per year, per seat, and feeling that I can build it in a reasonably short time frame, once I have written a bunch of macros. Mmmm. Nothing like a challenge to get me going =)

It's knowledge, not intelligence.

First off, an admission. I'm probably the dumbest of my clan. I have next to no artistic ability, i'm not musically inclined, and I don't have the razor sharp wit. In short, I'm the runt of the litter.

In some ways, knowing I'm not the smartest has been an advantage. I've kept at things long after my other siblings gave up. I'm nowhere near Gladwell's 10,000 hours for mastery of any craft, but I've kept at computers long enough to be able to get things done.

It's why I'm sad about my brother. He's as smart, if not smarter, than I am. Yet he has wasted his talents by being combative with people who were trying to help him. He was so busy trying to prove his smarts that he never had time enough to pick up the knowledge.

In short, his insecurity driven rampages damaged the very friendships upon which his success was reliant.

My head hurts.

I've just spent three days trying to understand Clojure's for macro. And I reckon I have another couple of days worth of work pulling it apart yet.

It's a bit of a disservice calling it for, as it is actually a list comprehension, but it is called for because this is the construct to use when a traditional programmer would reach for a for loop.

It has iteration, variable binding, and choice. It is actually a baby programming language in a 30 odd line macro. I can see the parallels between this macro and the de-sugaring that haskell does with the monadic do blocks.

But still, it is a mind warping exercise pulling apart this macro. Fun tho. =)

Convergence

Clojure, a Lisp for the JVM with STM and Actors. Bringing the power of macros and functional programming to concurrency.

Terracotta, a network attached memory for clustering JVMs and persisting objects across JVM restarts.

jQuery, a JavaScript native api for manipulating the DOM using CSS3 selectors.

Raphael, a JavaScript library for creating vector graphics on the web, using SVG on Firefox, WebKit and Opera, and VML on Internet Explorer.

GMail outage...

A couple of days ago there was a GMail outage. Short summary, a combination of taking down a data center and a bug in the load distribution code leads to overloaded data center, and thus a bunch of people can't get to their mail.

A friend ribbed me about the readyness of Cloud Computing, on the back of this outage. My initial reaction was to defend Google on uptime stats alone. Now that Google have spread the love and admitted the cause, I'm going out on a limb and stating that very few (if any) corporate IT shops are prepared for dealing with entire data centers being taken off line for maintenance.

I mean, sure, there are corporates who have back up data centers, and even have machines in stand by DR mode. I'd be exceedingly surprised, however, if everything actually worked. Most DR tests I've seen were epic fails, and the fun thing is that DR readyness tests usually happen once a year, if at all.

If a corporate wants DR to work, it has to be a regular thing. Schedule DR cut over every friday night, and keep repeating until it works flawlessly. And then bring it back every month to make sure it keeps working flawlessly. Otherwise the business as usual is going to be in for a shock when things go pear shaped for real.

System integration nightmares

I'm currently poking my way through an enterprise service bus, figuring out how to make it do stuff. I've just successfully written my most complicated hello world program to date. It includes message formatters, message routers, two seperate processing pipelines, and it's processed across a SOAP interface generated from a generated WSDL file. Over engineering for the win, right?

The underlying metaphor embedded in the configuration language is one of a pipeline. Pipe line stages can have filters on both input and output, the output stage can select the next stage depending on attributes of the message, and the center of the stage, the component, does some form of transformation, all the while being isolated from the message transformation and routing happening around it.

The issue I'm having with this metaphor is that pipe lines are not turing complete. I need the ability to take a message in over a SOAP connection, do a bunch of steps (including writing to our local database as well as querying and updating a pair of remote ERP systems), and then return a result over that said same SOAP connection.

Now, in the simple examples, this return can be done, because an input endpoint can have metaphor breaking property of being inout as opposed to just in. What I'm scrabbling through the documentation attempting to understand is how i pass this SOAP connection through the chain so that I can return the result, even after a long trail of transforms. Not sure if it's going to work, or if I'm just going to have to cheat...

The most interesting point to me is that the pipeline metaphor is insufficient.

Thoughts...

Dad told me a story, a long time ago. It goes something like this.

Ted, a manager is asked by the CEO to come to dinner to discuss the possibility of stepping up to divisional chief. So on the appointed day, Ted clocks off after a day of fighting fires, goes home, gets changed, and heads in for dinner with the CEO.

On the way in, there is a terrible accident in a tunnel, and Ted is stuck in instant gridlock. (This story dates from the days before mobile phones, go with me on this one.) So after fighting his way through the accident debris, he arrives at his dinner appointment only an hour late. After explaining why he was late, due to the tragic circumstances in the tunnel, Ted was promptly told by the CEO that he wouldn't be getting the Divisional Chief position.

At the time I thought the CEO was being overly harsh on Ted, given that the circumstances surrounding Ted being late were outside of Ted's control. It is only now, looking back, that I understand the CEO's point of view.

I'm increasingly seeing that people split into one of two pretty neatly divided categories. Those who do things, and those whom things happen to. The language used by the two groups are largely not interoperable, let alone expectations over behaviour.

Odd, but true.

Code organisation...

There's a Law floating around somewhere about how compilers will have as many passes as the organisation that is creating it has groups. In short, the construction of the software is based on the communication fault lines of it's creator.

Traditional web apps have similar break downs, there are the sysadmins who wear metal shirts and have long hair, the database admins who are uptight and wear business shirts and demand respect for their models, the webapp tier guys who wear business shirts badly, and the web devs who wear jeans and indie tshirts, and frequently don't make it in on mondays.

These groups hate each other with a passion, and it shows in the application.

With the upcoming change of pushing the database out into the web client, and the back end becoming nothing but a persistent change notification hub, all the skills these current groups have in isolation need to be all brought to bear in the front end client.

By now I'm resigned to the fact that we are going to lose a generation of developers over this. We have on each other major change or technology. It's funny, but it took talking about all this at length with Dad to realise that we have seen all this before, and we will see it again.

A generated future

One of the things I like about Lisp is the fact that having a powerful macro capability in the language allows for concision in the code base that isn't available any other way. Unfortunately this leads to programmers minds exploding - powerful macros take time and patience to grok.

Near as I can tell, the other option for handling complexity also leads to programmer minds to explode. Instead of having an in language macro capability, I'm winding up generating code. The generated code, given it's weight in LoC has a similar mind bending capacity as Lisp macros, but the generated code is what is checked into revision control.

Is there a way out of this?

Antlr, a reflection

I worked with ANTLR, attempting to build a small code generator. ANTLR, along with TextTemplate includes a complete toolchain for building everything required for a code generator, ANTLR includes both a parser generator and a tree walker generator, and TextTemplate is a full templating solution.

However, working with the ANTLR suite turned out to be harder than I was expecting. Debugging the generated AST transformation code required a lot more spelunking through generated code than I was happy with. ANTLR has good support for debugging parsers over text, but I couldn't get debugging tree walkers in the ANTLR tools working to my satisfaction.

I can see two alternate paths going forward. Either I take the lisp approach and build out tree transformation code using something like clojure, and learn to love emacs and clojure's REPL. Or I take the path of writing code in a programming language written by compiler writers to write compilers in. Two obvious examples include Haskell and Scala.

The reason I am persisting with figuring this out is that we are on the cusp of one of those interesting times when we get to re-write the world again. We re-wrote everything when PCs appeared, then we re-wrote it again when Windows beat out MS-DOS, then we re-wrote everything again when the web first appeared.

And we are going to do it again to allow people to be able to use applications inside their web browsers, disconnected from the web. This is different from traditional desktop applications, even though from the users perspective they appear to be the same. The new functionality is that data is being kept both on the client, in a web browser hosted sql datastore, as well as on the server.

Which means we are about to re-write all of the applications that business users use. Oh goody =)

Google's approaches to concurrency

I have a bit of a thing for concurrency. It is one of those challenges where in the answer actually lies in choosing the right collection of technologies such that programmers can achieve what they want to, safely and effectively.

The traditional approach, using locks et al, requires a level of mathematical proof that is infeasible. If we are going to use locks, we need to do it in a way that the compilers and run times will be able to verify that we aren't being stupid, otherwise it becomes too hard.

Google's various approaches to utilising concurrency safely are intriguing. The first, MapReduce, is beautiful in the way that it elides concurrency. Suddenly programmers can write programs that will be run across thousands of machines, in a way that is both safe and manageable.

Then there is the message passing paradigm that has surfaced in both Google Gears and in Android. This can be looked at as another implementation of the Actor model, but in both cases using languages that are mutable and in widespread use. It is the fact that messages are serialised across the communication boundary that gives safety, but the rest of the language environment isn't impacted. It only requires programmers to treat talking to another thread the same way that they talk to another process across the internet, a widely understood abstraction.

Where this gets interesting for me is the Chrome multi-process architecture. Here is an architecture that facilitates traditional programmers moving into concurrent code using message passing, but also in a way that is compatible with all the current libraries.

Actually, where it gets really interesting is Google's native x86 web plugin. This technique could be used to embed plugins as completely managed threads inside of an application. Near as I can tell this is bringing Chrome's multi-process architecture into other web browsers, by the back door.

What I want to know is what would happen if someone took Google's x86 plugin architecture and bolted it into something like Qt, where the thread responsible for the screen paint, the thread holding the application code, and various other threads (network io, database io, et al) were each one of these managed x86 processes? It would allow Qt applications to dynamically download code into a running application, and spin them up as additional sandboxed services.

The possibilities...

Mmmm

Feeling the rush, and calm. I like.

48 hours

That was a rush. Intense. Insane. I want more. I want time to think. I want to understand. I want to just feel. Body longing, mind screaming. Sanity yet prevails, almost.

Where to from here? I know not. It is in the hands of the fates.

*wolfish grin*

It's all about concurrency

Probably the most amusing, to me at least, comment after the AppEngine talks yesterday was "It's all about concurrency."

The thing with AppEngine is that it is factored in such a way as to allow insane amounts of concurrency in a way that doesn't break programmer's heads. The traditional programming by mutation based computation model is still present. There is no need for programmers to get side affect free religion to use AppEngine's massive scale out capacity.

The only downside of AppEngine is that it requires programmers to grok DataStore. DataStore is where the concurrency rubber hits the road, and it breaks programmers. Programmers are control freaks by nature, and being forced to give up control for MT safety is emotionally hard.

It's going to be an interesting decade. Just as the wide spread adoption of GUIs forced OOP on a generation of programmers, Multicores are going to force concurrency on this generation. I'm kinda expecting a lot of programmers not to make the transition...

Looking forward...

I'm looking forward to being closer to the sea.

Brain full now, check back later

Possibly the most amusing part of playing with technology is how routinely I wind up feeling completely and utterly stupid.

Change is in the air...

(And not just the smells of summer)

When the luminaries of an industry are all out on the hustings evangelising change, and by change I'm talking about stuff that no one has tools or methodologies for yet, then it's fairly obvious things are in a state of flux.

This change can probably be best described as the son of Ajax. The amusing thing with the explosion of Ajax was that it wasn't anything particularly new, most of the underlying technologies had been in place for the best part of a decade. What Jesse James Garret did when he launched the Ajax meme was give us the lexicon to talk about the changes we were witnessing with Google Maps and GMail.

These new web application capabilities forever changed the perceived capabilities of web deployed applications. Suddenly, every application being launched onto the web had to have these capabilities. Most back end coders scoffed, Javascript has long been derided as the joke programming language. But, what customers want, customers get.

This new revolution is going to change everything because the web browser is going to become host to complete applications, and the back end is going to become nothing more than a persistent communications hub for people to interact through. This changes the performance requirements of the back end, away from relational databases and towards appengine style distributed datastores.

The new capability that this gives customers is the ability to use the webapp from a disconnected client. This might not seem like a big thing, until you take into consideration the explosion in sub notebook computing devices, netbooks, iphones and androids, all of which are partially connected to the net through the 3G telephone network and the occasional WiFi hotspot.

This new capability is going to be a huge driver, and it requires a complete rethink of how we build web applications.

It's a tad freaky doing research, and getting results that include mailing list discussions in which I took part, all too long ago. It's almost like I'm retracing my steps, but I don't remember making them...

Odd

At the time, I knew I was fighting for a life. I didn't intuit that the life I was fighting for was mine.

In retrospect, I can see how I was controlling so many factors, it truly was an onerous and thankless task. In fact, after the event, I doubt any of the players will even think I had anything to do with what went down. But then, said actors aren't exactly known for their ability to step outside of the here and now, or even outside of themselves. The chance of any of them glimpsing the full picture really is nil.

I suppose that is why I feel alone a lot of the time, not for lack of company, but for lack of people who can see what I see. My father can, but mainly if i try to explain what i see all i get is blank of disbelieving looks. I learned long ago not to explain.

And now, finally, I have learned that I can walk away. I don't need to save people. In fact, they need to save themselves. I need all my power just to save me.

Dave Thomas inspired this one.

I lost faith in AppEngine, partly because of a pratt in England that annoyed the shit out of me, and partly because I couldn't see how to implement the applications I wanted using it. The DataStore, simply, wasn't enough.

Tonight I watched, horror struck, as Dave Thomas (Of OTI fame) told a room full of Java coders that Java is the new Cobol, and that interesting applications of the future are going to be written on the Cloud, and in JavaScript. Cue deafening silence as a hundred odd java programmers start burrowing under their chairs.

It inspired me. Instead of trying to use AppEngine's DataStore as a transactional store, which is something it sucks at, I can instead using it as a pub/sub notification hub.

This would move the query side out to the clients, with their local Gears datastores, and make updates the only part that had to be propagated from the client back up to GAE, and thence out to all the other clients.

It actually works. A complete application deployment platform built from Google's software. Kinda scared right now.

Craig Mundie on the Future of MS

Wharton interviews MS's Craig Mundie

Craig is predicting a future where in we have full blown client side applications that interact with the cloud. Migrating applications out of the web browser, if you will. The destruction of the web browser has actually been a long term aim of MS, as made obvious in the their post IE6 destruction of the IE development team.

Obviously, migrating people from web browsers - which are inherently operating system agnostic - back to client applications which are tied to specific operating systems would be good for continuing MS's Operating System monopoly.

There are application domains that are ill suited to current web browsers, ones that are compute intensive or graphics intensive, for instance. Games, primarily. However, the text rendering engine inside of modern web browsers leaves the current generation of application toolkits (Swing, SWT, WF, MFC+, Qt) for dead.

The biggest disadvantage to me of a heavy client approach to interacting with the web is that suddenly which computer I am sitting at matters again. Having two laptops and a desktop, that hurts. Having an application that comes in both web and heavy versions, and stores all configuration in the web would reduce that burden.

Soon, I suspect, I will need an iPhone version as well...

Right. That's it.

I'm taking a holiday. I'm stuffed. I've had it. Worn out. Tired. Roll me over, i'm friggen done.

Just looked at some positions that a recruiter sent me. I should have been jumping up and down, there is some cool stuff in there. But all I can think to do is to get some grub, drink some beer, and sleep. For about 48 hours.

Odd realisation of the day

I've just come to the realisation that my issue with learning new stuff is dealing with the emotional distress of going through the acknowledgement of incompetence. It's yet another grieving process.

I've seen the emotional response in others when attempting to grapple with something new, I saw it all over the AppEngine list, the Clojure list, and various other places. What is new is the realisation that my own aversion to study is linked to this emotional reaction as well.

So being able to manage the grieving means I can start learning properly. It's been a while...

I feel like i've fallen back in time...

Playing around with MS's Visual Studio Express editions is a bit of a time warp. It reminds me of when I was programming against win16, and I use the word against advisedly, back probably the better part of two decades ago.

MS obviously builds some of the best tooling for first use. Click here, fill in code there, and voila, one working web application. I'd like to say that the traditional J2EE stack approach is inherently better than this, but it aint. From a pure get stuff done, the bang per buck measure, C# ASP.Net with it's databound controls and integrated JQuery has the Java world up against a wall. Sure, VS licenses aren't cheap, but what is the cost of a team of Java developers?

This is going to be an interesting adventure.

An experience like no other...

I must say, today was an experience in overload. In many different ways.

It was the Sydney Bar Camp, and it was fun up to a point. When people started getting tired, man. Ranting about how Google is evil, then about how to verify who should have the power to verify if companies handling of data is valid, and then finally the whole australian net censorship discussion. Gar. Raw emotions on display.

My head hurts.

On the upside, I can see a bunch of new technologies stepping up to allow us to step through the multi-core insanity. F# on .Net is going to be interesting, to say the least. =)

Thoughts on Remix culture

One of the things that I like about developing for the web is that there is a culture of openness about web development techniques. There has to be, all the source is out in the open already. In fact, there is significant pressure on web developers to write good code because it's all hanging in the breeze.

One of the impacts of this is rapid migration of new ideas. Part of this is down to the ease with which one can pop open firebug and have a poke around inside someone's html and css. It makes for easy remixing.

I suspect this is probably the hardest part for adoption of Flash and Flex - all the really cool apps are not open for review and learning. Actually, the same can easily be said of back end development. *cough* Google *cough*

Realisation of the day

(Although, I'm sure I've had this one before.)

Application development is broken. This much is obvious to most folk. The reason is interesting in that it reflects something that is true of all design.

There is no right time to design the application's interface.

During initial specification, when most people spec an interface, most people can't visualise using the application. Thus you get underspecified and inconsistently specified interfaces. By the time an application is built, it is too late to specify, as change at this point usually requires complete rewrites of the majority of the code base.

In short, by the time people can actually use an application, their reaction is always "That's not what I want!"

There has to be a better way. And I intend to build it.

Change in perspective...

Possibly the biggest change after coming to grips with ANTLR is now no system is "too big" to tackle. If I can design a DSL that encompasses the design, then I can implement the whole thing. Sure, there is a whole bunch of work writing the various traversals and the code generation templates.

But there is suddenly no boring trudge work left. =)

Random musings

For an industry whose primary sales schtick is "money savings through automation" we are remarkably attached to doing things manually.

Heretical stance of the day.

I love of all things Google - I use GMail, Google Docs, Blogger, Analytics, AppEngine, and GWT. But that doesn't stop me thinking things are wrong with the big G.

One of the important things about capitalism is it's darwinian nature. One of the things wrong with most companies is that they are, for all intents and purposes, command economies. Google tries very hard to allow darwinian nature in, but there is a serious problem.

Google is the TRS-80 of the cloud era. It is amazing that it works, and it is creating a new world, but that doesn't mean it is of the new world. The TRS-80 was no Apple PowerBook.

Google is the leading edge, and thus being technically right is important, but for what comes next, design is more important than algorithms. So, who is going to be the Apple of the Cloud Computing era?

My head, it hurts...

CSS, JavaScript, Yahoo UI, ANTLR...

One of these days I might learn to not try to boil the ocean.

Nah.

ANTLR as a gateway drug to lisp?

I'm reviewing the code I wrote for my code generator, and it wound up looking a lot like lisp. Actually, a lisp with destructing bind. Like Clojure, say.

The biggest advantage for me with using ANTLR over the lisps is that I can, and do, look at the generated parser and tree parser code. I understand the underlying implementation, which is essential for being able to debug. Clojure is a step into the wilds for me, I don't really understand what it is doing, and there is only so much macro-expand tells me.

Hopefully playing more with ANTLR will allow me to understand how Clojure works...

Drunk, reflective, and in need of a holiday

I'm making bad decisions, being pig headed when i should roll over and admit i'm wrong.

A fair bit of it of late.

On the up side, i'm having a holiday soon. It'll being a juggle financially, but on the upside I'll focus on my projects instead of trying to force my work into being interesting.

And if my projects work out, work will get a whole lot easier, and maybe even more interesting.

This has been brought to you by the letters b, e, e, and r. And the number 2. Yeah, time I slept. More beer tomorrow.

Another direction change

Now that my code gen is starting to spit out code, and actually usable code at that, I am starting to get larger aspirations.

One of the continuing bugbears of my professional existance is the creation of webapps. They have a bunch of inputs, there is some processing, and a bunch of outputs. In all honesty, it sounds, at the highest level, like something that should respond to automation.

In fact, a lot of people have tried. The biggest issue is that all the current code generators are finished products. They are designed from the ground up to produce finished output. I think this is the wrong approach - the resultant webapps are invariably broken by the inability of the user to express their specific requirements in the tool's input medium.

I think ANTLR's approach to building a DSL, and then allowing embedding of chunks of java through it is the correct approach. The DSL expands, and the embedded java is exposed in place. I think this is what is required. A DSL for capturing web applications, along with the ability to embed java in it as the "business logic".

Mmmm. Yea yea. I've sold my soul to ANTLR. =)

Thinking about libraries in lisp

One of the things that always confused me with lisp is that for a fifty year old language, there is bugger all in the way of battle hardened libraries. Take java, a mere puppy at ten, and yet it has more libraries than you can poke a stick at. And don't get me started about c. We'd be here until Christmas. The year after next.

Now that I am playing with ANTLR, I think I have understood why there are no libraries. You see, the desire for libraries is to reduce the programming effort to achieve a goal. The amount of code required to write all the database access layer in a reasonable sized webapp is considerable, so people use Hibernate instead. No one likes Hibernate, in fact there are rants upon rants about how bad it is, but all said and done, using hibernate is better than not shipping at all.

The problem with Hibernate and co is that the programmer winds up giving up control. He (or she) has no choice but to interact with the library on it's terms. If you don't like the terms, find another library or write your own. If you want to do something like what Hibernate does, but slightly different? Fork it, if you are brave. More likely, you will curb your tongue, and agree that Hibernate's way is the one and only way, and deal.

With a code generator, this is suddenly not a trade off I have to make. As long as I can write prototype code from the database up to the user interface layer, and I can express my design in some computable data format, then I can slice up the prototype code into templates, and generate out the whole application. Libraries suddenly are a source of headache, constriction, and bugs, instead of being a saviour.

In other news, I'm finding Google less and less useful as a source of knowledge. Thanks to an explosion of indexed content of varied value - i'm thinking of web forums full of questions - my keyword searches are returning pages full of my keywords, but no actual usable information.

I'm resorting to long form book purchases to get the knowledge I need, often in pdf format. A positive thing for artists and authors, but worrying for the big G.

I'm becoming a religious zealot

(Becoming?)

Anyways.

I wrote the guts of a Model Driven Engineering code generator over the weekend. I think I might be getting ANTLR finally.

Only downside? Everything I look at I start working out how to build a compiler and/or a generator for it. Gah.

Getting one of the Zens of Lisp

It's odd, I've always admired, and been scared of, lisp. It's one of those languages that everyone always talks about emotionally. There are the zealots, preaching macros and homoiconic data / code representation, and there are the rest who pick on it's parenthetically obtuse nature.

I think I am starting to understand why Lisp's macro system is such a big thing. The amusing thing is that I'm not using Lisp to do it - I'm using a code generator to write a code generator from a hand designed data language that looks suspiciously like JSON. But it's a tad more prescriptive than JSON. Honest.

I'm finally getting the religion on capturing requirements in computable format. Clients always change one of two things - the original spec, or how they want the whole product to look. The first requires only that the requirements be updated, while the later requires that the generator be updated. In neither case is a bulk change to the whole code base required.

Finally, the project implementation can be correctly factored. I like it, a lot.

Constraints

The current webapp designs use a layered model, like all things in computer science. Each layer has it's primary concerns, the database layer is concerned with persistance, the client layer with presentation, the middle layer with everything else. The problem with this design is that abstractions leak, and constraints of each layer are visible in other layers.

My belief is that the correct way to engineer a webapp is to acknowledge the constraints of each part. The disks can deal with about 100 seeks per second per spindle, give or take. The channel between the web browser and the web app is high latency. And the most important is that any webapp design that requires disk seeks on either queries or updates is inherently DOSable.

This leads me towards a asynchronously updating network design, wherein browsers notify the webapp of changes, the webapp responds with changes it has seen, and the webapp streams updated state to disk. Think of a spider with it's eight legs spread on different strands, listening to it's web. The trick is going to be generating consistent code across the various nodes, the html, the javascript, the concurrent server code, the reporting code.

A new place to rant and ramble, a new face to see the world with, a new time to create new realities. As they say, growth is change, and change involves letting things go. 

And with this, I start anew.