Editra ... the good and the bad. Does the search end here ?

One of a great find that I found via my search and in the comments in my blog is the Editra editor. This editor has been around for sometime (2005 if the website's timeline is anything to go by). Editra is mainly a one man show namely a Mr. Cody Precord. Editra can be installed by using most Linux packages, Mac or even using the trusted: 'pip install'.

 Few of the things that I looked for in my evaulation of IDEs:

1. Easy / Right installer ?

2. Ease of configuration ?

3. Python tools ? Ease of use ?

I would say that Editra succeeds in 1 and 2 but falters at 3. What I like.. it's light and reponsive and do need a farm of servers' resources to run. The editor is easy to configure and most of the configuration item / controls are in the places where you would expect them to be. The layout of the window are intuitive and where you expect them to be and it's evident that this IDE really lives up to the hype "Created for programmers by a programmer". Heck, some so called "enterprise" grade editors controls and configurations are just funky and would take you half a day just to figure out where they are, so this in itself is an achievement worth mentioning. Well now that all the praises and niceties are out of the way, now let's get down to the grease and dirt to the unfortunate part of why Editra still could not wedge out PyCharm or Sublime Text 2 as my main work horse editor(s).

One of the thing that an IDE I feel should help me out with is the Source Control component of the code that I am working on. The source control section for Editra is a bit dodgy and takes a few run overs to figure out. So far the editor which handles this the way I like the most is Sublime Text 2. Something however must really be done on how Editra handles VCS files. Sometimes it would work and sometimes it would not. Although I checked out the whole source tree from the same repository, some files were versioned and some were not ? Up until today I am still trying to figure out how to view changes of changed file on the VCS. I even tried the latest version from the Editra Plugins repository but no cigar. It would just sit there and not do my bidding, worse it did not even spit out any errors which I could get a handle on in hopes of fixing it. When I went to the editra channel on irc there was no one there except ChanServ and me, so no help there too. According to the main editra site a Mr. Kevin D. Smith was responsible for the projects plugin which the VCS is part of, so probably I should try to contact him ?

I tried just about everything including the 'make patch' menu item but nothing happened too and guys showing this kind of logging for the app:

[12:24:07][app][info] Going to sleep
[12:24:09][app][info] I'm Awake!!
[12:24:12][app][info] Going to sleep
[12:24:13][app][info] I'm Awake!!

I can dig that this is supposed to be a programmer's ide but this is not really what I would call informative or helpful!

Snippets

As far as I can tell, Editra does not currently have snippet support or allows you to create/add your own snippet. This powerful feature can be found in PyCharm and Sublime Text 2 and is something I used daily.  This is a big downer as mastering snippets would really reduce the number of keystrokes you have to do daily as well as get your typing speed up to your thoughts (well almost).

In conclusion, I can only say that I just feel the most frustrated about this editor. Not that it's lousy just that it has so much potential but all of it lies unrealized.

Growing our bootstrap script

Our company currently is using a custom bootstrap script to install our application which is based on virtualenv. Our 'bootstrapper' does a few things:

1. Sets up virtualenv
2. Runs configuration scripts
3. Other stuff

While our script works and has served us well, when cleaning it up and trying to expand it's functionality it began to show it's cruftiness, the kind of whiff you get that maybe the script itself needs the help of some third party libraries. Two of the python libraries that I have seen that can help us are:

• virtualenv-tools to help us with cloning our virtualenvs
• unipath or py-path to clean up the parts of the code relying on os.path and gang (ugh really hate those)

It seemed simple enough initially, just add these libraries or third party applications and ensure that we install them on our live servers as well. So, there I was a confident idiot, accessing our production servers installing the required pieces, checking in the modified code ... going home for the day thinking that all was well. Well, all was not good and dandy. The next day my QA department looked at me with this crossed look because all of her test servers broke. Well of course they broke, they too used the same bootstrap script and they too needed to be installed with the new required pieces. "There must be a better way of doing this" I silently thought to myself, then I remembered testing out this application building system from the Zope foundation called "buildout". I wondered if it would be useful for me. I read through the docs and it seems it was just what I needed!

After numerous false starts (The acquaintance process took longer than I anticipated as grasping it's core concepts was not as easy as I thought). So now here is how I do our bootstrapping using buildout, I create a buildout.cfg with barebone contents:

[buildout]
parts = cot

[cot]
recipe = zc.recipe.egg
interpreter = python
eggs =
    virtualenv-tools
    virtualenv
    configobj

Then I run

python bootstrap.py -d

then to finally pull in my bootstrap's library requirements in the form of local eggs.

bin/bootstrap

In this way it's self contained and I need not touch any of the system's python's libraries. Nice! Finally I can run my bootstrapper with:

bin/python mysuperbootstrapper.py

Then after the bootstrapper runs it's course, I can activate my virtualenv and hand the reigns there. No need for me to manually go in and ensure that all of my bootstrap's system requirements of python libraries are met. Done!

There are few places where this workflow can improve, one that comes right off the bat is why not just make the whole thing a buildout process. Well, I am looking into that as there are more than a few places where our setup.py files has to be rethought and better-ed. I am looking at that and hoping to reshape our whole deployment plus bootstrapping process hopefully based on buildout. I am also thinking of reshaping our testing process around buildout to get rid of retarded stuff like installing nosetests on production servers.

I will post my findings on our progress in the migration process ... as for now I am complete.

What Sublime Text 2 got right.

After pondering on the IDE matter more, the two finalist in my mind that are worth considering for now are PyCharm and Sublime. This week though I just found myself using Sublime more and more instead of using PyCharm. I am not saying that PyCharm is bad or in anyway less than Sublime. I guess I just find Sublime able to fit into my mind easier. 

I began pondering why in a relatively short time Sublime was able to gain such a foothold as well as mindshare in the developer's community. Just search in youtube or google for Sublime vs PyCharm to see what I mean. I feel that Will Bond of Sublime just did a few things right in my opinion.

1. Kept performance of the editor as it's main goal firmly in sight. I just get the feeling that performance always trumped functionality if there was ever a toss up during the development of Sublime. Never once when I used this editor did it feel bloated or sluggish.

2. Sublime just remembered what Nokia realized all those years back when they launched and marketted their multi color phones in the world of drab black color 'professional phones'. Sublime remembers that although the editor is made on a machine that it's main / only audience is a human. Sublime's out of the box colors / fonts and settings until today still remains the most eye pleasing out of the box and normally I just leave it at the default settings.

3. These two things in turn created the third thing in which Sublime gets right which is having an eye pleasing and easy to use forum where the users searching through the knowledge base can get to the information that they are looking for easy and quickly.

4. Create innovative new features such as their right hand code map which I find really useful for moving around while at the same time giving a nod to the old true and tried ways... read Vintage mode. The fact that Vintage mode is built in and not some third party package also gets a big plus from me. For those who don't know Vintage mode is Sublime speak for vim keybindings.

5. Have some pretty informative videos teaching the usage of sublime. Have a look at this Sublime porn (Although it's a bit targeted towards web developers) and tell me are not raring to get on the Sublime band wagon. The guy can really work fast!

6. Allowed and let flourish the tools / nuances of different languages so that developers of any language can just feel at home on Sublime.

I agree totally with people who say that Sublime was created by a developer ... in this case Will Bond for developers. You can feel that Sublime was created trying to scratch some of the itch he himself was trying to scratch and that makes a really powerful product. At this juncture though, I just discovered that certain really nice to use plugins, such as SVN plugin requires you to pay after sometime. While you can always just choose not to use it, but then I prefer it if all of these plugins were just included as part of the main install and just charge me upfront.

Also, I like the fact that Will and gang have developed this so-called 'App Center' by accepting and creating a repo for the community to contribute towards the development of plugins for Sublime. There are paid / free plugins available but seeing as how you can (for now) just keep on using indefinitely the paid plugins I guess you can decide to pay / or not for the plugins that you like. Personally I have paid for Sublime, the SFTP and SVN plugins all by Will. I really hope that Will will walk a non-Apple path by allowing healthy competition if his users decide to come out with free packages that will compete with his commercial ones. 

I will in the end have a look at PyCharm and seriously use it for at least a week to see if I will eventually pay for it, but compared to my first time on Sublime, I just found myself fumbling around more looking for stuff on PyCharm or just stopping my work sometime waiting for some unscheduled index to finish running. As for now it looks as though Sublime as a lead over PyCharm.

State of Python IDEs 2012

The search for a Python IDEs is like for lack of a better analogy like masturbating. Every serious programmer or serious programmer wannabe, more so the novice programmers have done it before or at least thought about it once or twice before. I know it's a bad analogy but you should know what I am trying to get at. We have all done it before one time or another or at least thought hard if there is something outside there better.

The macho among us will claim that we only use the true and tried Vim / Emacs but given an IDE that can do all that both of those can do and more would you not be tempted ? Some would argue that it's not about the tool and we if we put the time into it we can customize something usable out of the free stuff that is available out there. I for one would rather just pay for something like or just have everything available out of the box that is configurable without much effort and I am just a sucker for how good a great editor feels like when you are using it. At a certain point I can almost see the wisdom behind why people would rather just survive without plugins on a vim install ... but use vim without my NerdTree ?! Woe! What kind of life would that be.

Well my search went into serious mode for these past 2 weeks. I walk away from the experience a much frustrated but wiser man. Summary of my findings, which is quite shocking .... none of them are quite ready yet! I thought that this would be the state of Python IDEs say 2 to 3 years ago but it seems that the scenario has not changed much. Here are the few that I looked at:

1. Geany
2. Kate
3. Gedit
4. Spyder
5. Ninja-IDE
6. Anjuta with PyDev
7. Vim

2 and 3 are there because they are so ubiquitous with Linux distributions and I wanted to try them out to see if I can have an editor that comes with almost all Linux installations that I have not spent too much time with them and have something up and running in no time. Alas, finding that pep8 and pyflakes seemed more like a third party integration does not spark the hot fuzziness in me. that I can have out of the box without installing much of anything and just use them. Yes, I did look around and stuff like this for Kate, just did not work correctly either and even vim integration with Gedit looked aged and at last just did not work (Just spent another 30 mins with it to find the location of the plugins folder moving from gnome2 to gnome3 .. yeucchh).

Let's get to the so-called review part of this post. Let's start with the grand daddy of them all:

Vim

I have been using Vim for the longest time and to a certain degree it was great plugging in to my local pyflakes and pep8 to help with my Python code and the reason my love started to wander was because of the way plugins are managed. I used pathogen to manage my plugins and for sometime I was happy until strange errors started popping out here and there because some plugins were having errors. I found it quite hard to debug and found myself sitting at my terminal late one day deleting my plugins one by one until I finally arrived at the plugin that was causing the error. I patched it up and sent the author the patch, yet the whole experience created the seed of thought in my head that there must be a better way of doing things or at least a better editor / IDE out there that can be used.

Truth be told vundle seems promising and after knocking it around a bit it seems like a good way to manage your vim plugins across machines. Read this ... http://www.charlietanksley.net/philtex/sane-vim-plugin-management/

Sublime Text 2

This is the editor that I am still using today. While initially put off by it's text based config / settings files I stuck with it and was happy as it was light and responsive and it had everything that I needed, except maybe a debugger or a prompt that came out of the box. I was so happy with it I bought it and till today it remains my main workhorse. About 3 weeks back something happen though that diminished my love for sublime. I installed a new sublime editor in another machine and pointed it's plugins at my Dropbox install. This way all of my favorite plugins will always be available to me. It was strange, it just hung there refusing to start up. When I deleted the symlink to my Dropbox it just started up normally. There was no error messages, no logs nothing indicating what was wrong. I was quite devastated as like many I hated to have to go and look for another IDE / Editor that I can use but I was really up my creek without a paddle, as googling for the phrase "Sublime can't start because of plugin problems" yielded nothing useful. The other thing that irked my was the fact that I had to lug my plugins around all around in my Dropbox folder. The issue that I had with Vim was not really solved, infact it became worse, because with vim at least it could start up, it just spat some angry error messages at me.

Actually it would be unfair of me to comment on Anjuta with PyDev, Spyder or Ninja-IDE as all of the these IDEs just could not install correctly on my box. Anjuta was just dogged slowed my machine down to a crawl that even firing up a browser to go and look at what was wrong was rendered impossible. Geany I just installed today, but once I found that vim bindings was a wishlist item rather than a configurable item, that really put me off. At this point I am really wondering if PyCharm or WingIDE would be worth the investment. I tried ver 1.0 of PyCharm before and my initial view of it was quite laggy and consumed resources on my machine. Probably would be good for me to try out the 30 day trials of Wing and PyCharm as they can be quite pricey... 99USD for Pycharm and 245USD for WingIDE.

What seems to be going for PyCharm now as I look at it are:

1. Guido seems to be behind it.
2. Some obscure functionality which I usually use which I find hard to find anywhere else like the ability to stash away changes. This is something that is very useful for me which I only find that I could do using git. This is really one of the most attractive points for me to consider getting a license from these guys as it allows me to do this outside of whatever VCS that I am using and I do not need to do it two times.
3. Integrated nosetests and coverage.
4. Integrated pep8/flakes checking for code. Been using these for sometime now and I find them indispensable.

Down side seems to be:

1. Based on Java, read dead slow plus resource hog. Related to this I found this ticket today which looks a bit worrying.

Actually in summary, I can see very little wrong with PyCharm except that it becoming a resource hog which usually is the case for IDEs written in Java, just look at Eclipse. Enabling and configuring PyCharm to get semblance of a working environment for myself was quite easy and the controls seem intuitive without much need to hit google for an answer. Just as I wrote this my whole desktop became unresponsive because of the resource eaten up by PyCharm. The jury is still out on this one. I will post what I eventually do in part 2 of this.

Presenting ... the inupypi server

After using the pypiserver: http://pypi.python.org/pypi/pypiserver/ for sometime to house our internal eggs and libraries, we (one of my colleagues at least, very very quietly I might add) decided add to give back to the Python community by developing our own pypiserver called InuPyPi: http://pypi.python.org/pypi/inupypi/0.2.3 which is based on Flask. Some of the features of our pypi server are:

• A nice(r) web interface for packages.
• Supports multiple eggbasket folder per server.
• Auto searching for updates of packages (planned)
• One click update for outdated packages (planned)
• Command-line tool to upload to server something like 'python setup.py upload' (planned. This will be pretty close to doing)

For those who are interested to contribute the project, the project lives in github: https://github.com/hallaj/inupypi. We will appreciate suggestions and help with our pypi server. 

How do you test your Python apps for production ?

As usual I am just thinking aloud for this post and thinking about the ways to solve a problem that we are having. Haven't really got the answer yet just sharing hoping that some other smart soul out there might offer me their 2 cents. Our apps currently have got pretty good test coverage or so I feel. I would say that our coverage currently is about 95% of our application functionality. We have unit tests, functional tests and even selenium tests (currently trying out splinter to replace selenium). We run them all against our test fixtures and use Jenkins to visualize and notify us whenever the tests fail. While I think there are more rooms for improvements, on the whole I am pretty proud of our whole test architecture.

We freeze our code and only branch our code once all of it's tests hits blue on Jenkins. That's all fine. Hunky Dory. That's when the ice berg reared it's ugly head on our sailing Titanic. The problem is our code failed twice while we were trying to deploy the app (wsgi app) and we only detected it through manual means, eye balling it. Once because our ssl certs failed because of this particular bug: http://code.google.com/p/httplib2/issues/detail?id=202#c2. This bug occurred with httplib2 when using wildcard certs which we don't use in our testing. The whole wsgi app came crumbling down without any of our tests informing us of the wiser. I would be okay if these were some obscure bug that occurred because of some strange use case that we have never considered prior to this, but these are big-pie-on-your-face kind of embarrassing show stoppers. What I want to do is a way to catch show stopper bugs before deploying our code on production servers.

What I am talking about here almost like a full dress rehearsal before deployment. Setting up a dedicated machine for testing that mirrors the live data seems the most obvious way but in someways is too much of a resource hog in my opinion. From what I gather from doing a bit of quick 2 minute research I am not alone:

• http://www.certpal.com/blogs/2010/06/testing-in-production/ and more recently from the Python Planet .. 
• http://www.alexconrad.org/2011/10/proxying-tests.html . This is a start but it still means almost mirroring my live site on to a test box.

This seems like an idea too http://xunitpatterns.com/Test%20Logic%20in%20Production.html. I think overall the main thing to be moving towards is to stream line the test machines to be as close to the live machines as possible and to lessen the 'magic' that we do with test servers to make tests work and determine which environmental variable would be crucial to our application being deployed well and replicating that in our tests. 

What I don't like about this solution is that for 'n live sites with different environment' you need n amount of replicated live environments. Another idea that I am playing around with is the create a suite of 'critical tests' that is run after an upgrade just to ensure the most critical services are running fine. Ideally these tests should be short but it should give you an a pretty good idea if the main services will be running when you go home after an upgrade is performed on the server. The tests should just be a subset of the main battery of tests and should complete in a fraction of the time taken to run the full suite of test, just like one of those old windows installers that run 'self diagnosis' after installation. 

OpenSuSE 12.2 delivers again

Just upgraded OpenSuSE to 12.2 and I am happy to report the upgrade process is flawless as usual. Stuffed in the DVD, settled some packaging conflicts and I was good to go. Half an hour later 12.2 was smiling back at me after rebooting after I restarted my machine. This is by no means a review, just a little tip of the hat to the OpenSuSE guys for making the upgrade process smooth.

I don't think enough is said or enough accolades is paid to hardworking folks that helps me to retain a few more of my already receding follicles after a major upgrade. Thank you guys! Aside from having to re-install my google chrome and my codecs (Why can't they just have an option to do or enable that during upgrade process ??) nothing much of anything was interesting. Boring. Crickets. That's the way a smooth and great upgrade should be.

Really like the work put into the boot up screen and the improvements done to zypper. Overall the box at homes is chugging along at a somewhat higher speed.

Thank you OpenSuSE. Just the other day it crossed my mind that I have had more than a decade affair with OpenSuSE and it just seems get stronger from release to release.