I have been writing an Agile project planner, which is quite amusing! I have been, and still am, against the use of such tools (for another blog). Anyway, it is a good candidate for a web application and there is enough depth to give rails a decent trial.
So far I am quite impressed. The shear pace at which you can knock out functionality is astonishing. For the first time I think I am actually enjoying writing a web application. The amount of code that I have written is negligible, mostly view code and a few bits added to controllers etc. Largely it has been a straight forward and enjoyable exercise.
The next step is to exploit the built-in ajax and drag and drop effects to create a cool way of moving cards into an iteration and progressing them across a Story Board.
Once that is done, all that is left is to train my daughters up and start a web dev company. I'll get my daughters to pump out rails apps after school ;-)
Still lots to learn and I haven't had to deal with deployment, configuration, and other things that always give you headaches on most projects. Still not convinced on the Database-Driven approach either, feels a little odd. Also as it is easy to knock up apps quickly, it is also very easy to create a complete mess faster than ever before! However, thus far, given the choice between writing a web app in Rails, Java/.Net etc, I think i could do the development piece in rails much faster and have loads more fun doing it.
A question i have been asked a lot over the past 9 months: How do you do TDD in the world of Grid Computing
The way I do TDD is to push away external systems, libraries, components etc, by wrapping them with Interfaces. This gives me the abilitity to Test/Design the behaviour of an application without external dependencies having too much influence on the overall design.
With Unit level tests I can then substitute the Interface to the external system (Grid in this case) with a Mock or Stub implementation. This provides a number of advantages:
Of course this approach doesn't cover case where you really want to know that it works when everything is plumbed together. That is what Functional tests are for.
Functional tests should test from top to bottom with real dependencies plumbed in. Further, they should run as a seperate stage in your build pipeline so they don't slow down the feedback loop. Slow feedback == frustrated developers == lower moral == short cuts == a mess!
So the moral of the story: TDD on a grid application is no different to TDD in any application. Fail fast and be happy!
]]>Ok now that I have got that off my chest; why do I like Jini and in particular JavaSpaces? Because, at heart, I am a geek! Seriously now, Jini offers true network plug and play computing. It is built with the fact that networks and computers etc are unreliable. Networks, PCs, Services etc, can come and go, Jini caters for this, enabling you to build reliable and scalable Service Oriented Apps relatively easy. Throw JavaSpaces in to the mix and you get a nice simple way of de-coupling your Services.
With these technologies I can build a simple compute grid quite easily. Basically, create a Worker that pulls Entries from a JavaSpace and calls an execute method, start up 1 or more JavaSpaces, start up n workers, my simple compute grid is now up and running. Ok there is a little bit more to it than that, but you get the idea.
JavaSpaces is more than just an infrastructure for enabling Grid computing. It can be used for distributing data, de-coupling services or components (whatever you want to call them), or adding on-demand scalability to applications. I am sure there are plenty of other usages.
So what I am trying to says is; If you haven't already had a play with JavaSpaces, go and download Blitz, install it, have a play, and you might find that you are pleasantly surprised.
Last but not least; Sun - please please show some support for this technology!]]>
So, what is a Grid? That is an excellent question! Unfortunately there is no one size fits all definition for what a Grid is, it has a slightly different meaning depending on who you talk to. In its most basic form, a Grid is just a batch scheduler with a bunch of compute resources. Not very exciting I know!]]> Types of Grids
As the previous paragraph suggested there are a few different types of grids. Here i define those that I know about. Note some of these approaches may be used together.
Most people are familiar with the SETI@home project, it works by stealing CPU cycles from Computers attached to the internet. Well it is not exactly stealing! You download and install a screensaver on your PC, which when activated, i.e., your PC is idle, will connect to a server and download some data to analyze. So whilst your computer is idle it will steal CPU cyles from it, but as soon as you de-avctivate the screen saver the analysis stops. This is a very specific example, but the same approach can be used in a more general form. For example: You install a very small "agent" program on corporate desktops, that, when the cpu is idle, connect to a scheduler and asks for some work to do. The work performed by the agent could potentially be any arbitraty program. Of course there are environmental specifics such as Operating Systems etc to take into consideration.
Quite a lot of the grid products in the market are little more than batch schedulers. This type of Grid usually consists of some central scheduling/management server and a bunch of agents on dedicated compute resources. The programs executed on the agents are typically any executable program.
The batch scheduler grids I have looked at so far have a significant overhead associated with getting your work scheduled to run on the grid. This usually means that you need to have fairly long running tasks, and quite a few of them, before you actually see any benefit of using the grid. Of course this overhead leads to under utilization of the resources on the grid, which is a little alarming as most corporates want a grid to make better use of these resources.
Ah yes, SOA had to pop up somewhere! This is the way the grid standards are heading.
Suprisingly, at least to me, most of the grid products I have come across do not consider this approach. I have come across 2 products that have some sort of Service model, one of them is quite good where as the other one, well, it is the other one!
There still is some scheduling component as part of these grids, scheduling is core to any grid solution, but now you are invoking a method on a Java/.NET/C++ object instance rather than asking an executable to be run.Of course there are nice ways and not so nice ways to do this. To me the nice way means I can invoke arbitrary methods that take rich objects as parameters. The not so nice way means that everything has to be a string or byte array.
You'll notice that I haven't mentioned data once until now, but surely data access/transfer will quickly become an issue on any grid. The most common way of getting data about on grids seems to be shuffling files about. Shuffling files may be fine for slow changing data, but what about per request/task data? Also, I know that I really don't want to have a file system that is being hit from potentially thousands of compute nodes; bottle neck anyone?
One of the alternatives we have looked at is to turn the problem on its head and think about a grid from a data movement point of view. After all a scheduled task can really just be a few bits of data, i.e, invoke methodA on class B with paramaters c,d,e.. This lead us to look at JavaSpaces as an approach for building a Grid.
JavaSpaces is so simple, yet so powerful. Four very basic methods, read, write, take, and notify. With these four methods and a simple Master/Worker pattern, you can build a very basic grid in a couple of hours. Guess what? Unsuprisingly, JavaSpaces is pretty darn good at moving data about. It enables you to schedule tasks as well as move data about all with the same API. The main problem we have had with JavaSpaces exists on the resource allocation and management/admin side. The products, both commercial and OSS, currently have management only at the Space Entry level, we need more than this for managing potentially 100s of thousands of running jobs/tasks.
To me a grid is a completely dynamic compute utility that runs in an heterogenous environment. Heterogenous in terms of Hardware, Operating System, and programming language. The grid should at least support a SOA approach, and if needed, for legacy reasons, batch scheduling. It should also be clever about how data gets into and out of the grid, sorry but shuffling files about just doesn't cut it, i want a transactional cache. There should not be a central scheduler/manager, but rather a set of dynamically allocated resources for this, i.e., the grid uses the grid to run itself. If there has to be a central scheduler, please let it have more than one back up and please don't tell me I need big iron to run it.
There is a bunch of other stuff I haven't even touched on, i.e., dynamic allocation and SLAs etc. Next time!
Grids come in many different shapes and sizes, there is no one size fits all solution. In fact, most corporates probably want/need a mixture of the grid types mentioned above. Though i have been mostly underwhelmed by the products in the market place, this is nonetheless quite an interesting area to be working in.]]>