Sneak peak preview of upcoming book

My friends at packt has sent another book for me to review! The Grok-1.0 is up and I have been looking for this as I have been looking for just an excuse to get my elbows dirty up to Python oil again!

You can see a preview of the book here:

Preview

Even better a sample chapter here:

Chapter 5

This is going to be good!

Python training in Malaysia

A guy contacted me recently regarding training. Me tried of course to look for local trainers here but some were either too expensive or just strangely did not reply emails. So it boiled down to having to do it myself. My junior programmer (or should I say sys admin since he said he wants nothing to do with programming) asked essentially can we survive being a Python trainer in Malaysia. Sadly, the answer is still no. Training here are few and far apart. After hearing my answer, his face showed a kind of demeanor that your neighbor would have if you drove up the drive way with your broken down Toyota claiming that this is dream car.

He must have been thinking that Java as a language is still better than Python since he can at least see some Java jobs on jobstreet and stuff. Generally that is still the whole conception of Python in Malaysia. Talks do help. We just need more. At least nowadays when I talk to corporates about Python they don't ask me why I suddenly bring wildlife into a conversation when they asked me what would the ideal programming language that their IT department should adopt for high productivity. I should hit the university level more to create more awareness. I have thought classes for corporates who were full of Python doubters and at the end of the class, all the Java people walked out at least respecting Python for what it can do and actually contemplating using Python in their environment.

It's really gratifying to see programmers who were initially so skeptical about Python suddenly light up to the possibilities of using Python for their work after being in the Python training with me. That to me is priceless and I wish that I could do more of these kind of Python gigs.

I think that awareness is the key for us Pythonistas to be able to survive doing the thing that we love so much here. I think that here at least if more people knew about the existence of Python it would really help and then I can finally tell that doubting junior system administrator ... "Yes! There is a future being a Python trainer!"

Project Documentation

Currently I am in the process of passing down my project to some juniors and I find the need to document it. Me being the lazy ass programmer that I am obviously started to look for something that would allow me to complete my documentation in as least steps and most painless way as possible. Found two main applications that I am using now:

1. Sphinx -> For main documenting work.

2. Happydoc -> For parsing and generating code documentation.

Both are quite easy to use and allows documentation to be generated in various format, which includes latex, html or pdf. Nice thing is both supports rest type reStructured text formatting which I just absolutely adore and is great to work with. I am especially impressed with Sphinx and the beautiful documents generated by Sphinx, coming with it's own search and everything. It's just what the lazy programmer doctor ordered. I am still evaluating both of these programs and am still open to other choices but so far these fit the bill nicely.

django-evolution aka orm pain part 2

Finally bit the bullet and tried out a new application to help me do the changes I needed to do on the db and boy am I glad I did. django-evolution is a joy to use. It's easy to install with no strange dependencies and it did everything out of the box as described. Initially somethings did trip me up such as the django-evolution's initialization actually looks only at the models.py and does not actually bother what is really in the db, as in say the models.py you define a field that is not actually in the db during the first 'python manage.py syncdb' to create the django-evolution table, it does not actually know that and will just carry on not knowing that there are discrepancies between the models.py and the actual db.

After I figured that part out, it was easy. Just add the fields and their attributes, re-run an update script on my data and everything is done. Now my selects are much faster. Initially a 3k row db select was taking 50s and now using the caching functionality of select_related() it comes back in a blazing 4s. Ah here too initially I got tripped a bit as after putting in all the relationship I noticed that it still did not do a cache select_related until I did some digging and found that all columns defined with 'null=True' is not cached by select_related(). Looking at the actual sql generated by the Django CRM, it looks proper and very well done. I am still checking out the little nooks and crannies of the application but so far I give django-evolution the thumbs up! I am actually looking at the code to see if during the initial syncdb django-evolution can actually peer into the db to see if the definitions are actually kosher before proceeding.

When you play outside of django's ORM ... you get pain!

For one of my projects, I forgot to implement the relationship of one my models. I thought I could get away with it by writing a tag that made a query to the db upon being fed a string in the report. To my horror, this caused the report to take about 50seconds to generate for a paltry 3000 record db! Upon tailing the logs of query.log, I found that the multitude of query to the db was causing the problem. It was basically hitting the db about 3000+ times for a single page report. That is when I found out the good thing about Django's select_related(). The problem now is, I do not have that relationship and need to build that relationship into the db. Not exactly fun with a db that is already populated with data. Led me to thinking, while Django's ORM might be great for stuff like calling data from another table, but would it also incur a performance hit everytime data is being queried. I have heard Django being used for high traffic sites but then are these high traffic sites?

I tried looking around but I could not find a satisfactory answer to this: "What if you wanted to model data for an existing MyISAM table which does not have the relationship?" It would be nice if Django provided a mechanism to recreate the relationship between MyISAM tables that is implemented at the MiddleWare. Say if I were to go the Django way now, would I have to dump out the data, re-implement the tables in a 'Django-ic' model then reimport back my data?

Currently I am feeling that in order to play in Django ORM's park. I am being forced to play by it's rules. If I opt to write my sqls raw then the ORM takes back it's ball and refuses to let me even venture into it's side of the park and I am left to recreate back a lot of the convenient functionality that was provided by the ORM layer. There doesn't seem to be a nice middle ground and that just sucks.

Patching Fun - Update

Looking around for a solution for a cleaner way of patching up my code on the Windows server, without installing the unixtools for windows, I eventually arrived at the python-patch tool. This nice little (less than 600 lines) tool allows you to have patch like capability and all deliciously packed in a Python script. Thanks techtonik!

So now I can just packup the patch.py Python script together with my patch file, and Walla! the admin at the other side can just run something like:

python patch.py diff_patch

to patch up my code. With a little modification, the patch.py can even generate a results.log for me to review the patch process. If anything were to go south, I can just send the reverse patch.

Now .... should I just get the script to autorun .... hmmm...

Patching fun (on windows) 101

I have sent about 100 odd patches to a system I am working on and in this post I gather together some of the lessons I have gathered along this path. I write all of my patches in Python and after much hair pulling sessions over the phone with customers am happy to say the sacrificed hair did go along to help me have a epiphany of what would be the best way to help customers who are willing to help us apply patches along. You can say it's a collection of what to do and what not to do tested in real life project scenarios.

First off and of ultimate importance is to get into the right frame of mind before you write a patch to be sent to a customer. The frame of mind you should be in is ... remember that paying customers never ever want to help you apply patches so if you run across one that is willing (like in my case), thank Allah, Buddha, the Universe or whatever deity you pray to for your lucky break! Your (paying) customer does not owe you anything, in fact you are in their debt if they even entertain your patch request. Get in this frame and your chances of writing a successful patch will be increased two fold.

Okay, here are some dos for writing patches. These steps below are for systems that are not accessible remotely and a bit physically too far or demanding for your to travel to.

1. Always recreate (as much as possible) the environment in which to run the patch. This means that if your customer's site has a G Drive (windows) your machine in which you are creating your patch better have that. I have had some great time tracking down why a patch does not work when it magically died while trying to open up a log file on Drive G in a machine that only has C and D drives. This includes in the case of windows of using the same binaries of libraries that your code belongs to. Just recently I had a good time on the phone with a customer who claims that come functionality was not working whereas using we were both using the same script! He insisted that my script was not working while I kept on suspecting that the problem lies between the computer and the chair. Finally I found the problem was due to an outdated binary build that he was using. If having the same build is no longer possible, at least go through the changelog of whatever library you are using to get prepared for whatever problems that might arise.

2. Always have backups of each of your patching steps. In fact it's best to include a sort of un-patch patch just incase your patch goes south. The steps to backup the important files should be done by your script and should not depend on your customer. So something like 'Please backup and replace file bla-bla" is not recommended because how do you know what your customer is backing up the original file to. Here you are thinking that they backed it up to something like .bak and they backed it up to .bak. A lot of fun will ensue in the case if you have to write a revert script later. Step no. 2 can be summarized as your customer should do as least as possible in the patching process short of running a patch script.

3. Generate error and result log of your patch process and ask the good will of your customers to send those logs back to you. This will help in determining if your patch process is successful.

4. Describe in your email what should the customer see in the event that your script is successfully run. I find here the quickest and surest way is to include a screenshot of what they should see. Typing out the results is error prone and it's hard to describe what they should see plus do you think your customers got that much time to read a long drawn out email?

5. Test, test, test and test each of your steps at least 4 times before sending the patch out. No testing is too much in this kind of scenario especially if you are dealing with live data.

6. Always include a way in which to identify what build are you customers currently running. This could be as simple as a BUILD text file indicating what BUILD are they currently running or in the case of a web application, a build number nicely tucked away at the top left hand or right hand corner of the screen. Everyone of your patches should update this BUILD number. This is instrumental if you were on the phone with your customer and want them to help you identify how updated are their source code.

Hopefully this post will help me remember the things that I did right and those that I painfully learned not to do again. It would be great too if it can help you avoid those pot holes that have claimed the lives of a few of those hair on my head. Happy patching!