Why not PHP?

DISCLAIMER: If you use PHP for building God’s Kingdom, then more power to you. Sometimes you have to do your best with the tool you have been handed. (However, if you have the freedom to swap tools, you should consider trying Python for a future project and see if it’s a better tool.)

My former supervisor, while I was in missions full-time, was adamant about NOT using PHP. I recall him mock-spitting on the floor in distain when asked by a potential recruit about using PHP. At the time, I was shocked!–especially because I was worried that this reaction would turn off the potential recruit from joining missions.

The truth is that anyone seeking to join missions should be willing to work with whatever tools are in use. Also, those who are decision makers should steer the team towards using tools that balance easy-to-learn with fit-to-the-task. Django is the sweet spot for web development, IMHO.

So, as a former PHP fan-boy who now maintains a PHP website, I understand a lot of PHP’s problems. (So, we’re migrating it to Django.) I’ve had pretty strong convictions about this decision for a while now, but this article really cements it. If you’re wondering why NOT to learn/continue to use PHP, read this article:

http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-design/

Speed up PostgreSQL with Django tests

My tests were taking forever. OK, only about 251 seconds (4 minutes), but that’s forever when you only have 11 tests! (In this one app.)

A lot of people talk about how to speed up PostgreSQL, but here it is in a nutshell:

Turn off fsync and full_page_writes in postgresql.conf

Specifically, here’s how I did it on my Linx Mint 13 box:

1. Edit postgresql.conf.

$ sudo vim /etc/postgresql/9.1/main/postgresql.conf

Change these lines and saved:

fsync = off
full_page_writes = off

2. Restart postgresql.

$ sudo service postgresql restart

Now, when I run my 11 tests, it only takes 34 seconds. That’s 8x faster!

NOTE: Probably don’t do this on your Production box.

Secure bcfg2 web reports with apache2 HTTP authentication

The problem

bcfg2 web reports doesn’t require any authentication out-of-the-box. (As of today.) This means that anyone who knows the URL of your bcfg2 web reports can see (and manipulate?) your server and/or clients.

Possible solutions that I didn’t attempt

The settings.py file contains an AUTHORIZED_GROUP setting. It looks like this activates the NISAuth authentication backend. I have no idea what NIS is, so I’m moving on.

The settings.py also includes the standard Django authentication backend, so theoretically, you could hack the views.py and use the @login_required decorator. But I’m feeling lazy today and want an easier solution.

The simple solution: Apache2 HTTP Authentication

I only care about one user: myself. So, I created a password file, using this command:

 sudo htpasswd -c /etc/apache2/bcfg2-passwords myusername

Then, I added this block of code in /etc/apach2/conf.d/bcfg2.conf:

    <Location /bcfg2>
        AuthType Basic
        AuthName "Bcfg2 Web Reports"
        AuthBasicProvider file
        AuthUserFile /etc/apache2/bcgf2-passwords
        Require user myusername
    </Location>

Disclosure

I did this with:

• Ubuntu 12.04.2 LTS, Precise Pangolin
• 64-bit architecture
• on Rackspace Cloud
• Bcfg2 Version 1.2.2
• HTTPS enabled on my server (don’t forget to do this or your password will be passed in cleartext!)

I also followed these instructions to install bcfg2-web (after installing bcfg2):

http://docs.bcfg2.org/appendix/guides/web-reports-install.html#appendix-guides-web-reports-install

Thanks to http://stackoverflow.com/questions/8417810/deploying-a-django-app-on-apache-mod-wsgi-with-http-auth

Also, thanks to solj and scofflaw on the #bcfg2 IRC channel.

Pitfalls in Django settings

Here are some common pitfalls to avoid when working with Django settings in large projects:

Pitfall #1. One Settings File for All Environments

Your Production server will have different settings from your local development, your QA box, and your team-mate’s local development. You could try to stuff every possibility into one settings file and switch based on environment variables, domain name, or other magical means. However, pretty soon this will become unmanageable. If your settings file has “if/then” statements, then you’re probably doing this.

Instead, you should break out multiple settings files for your various environments. A good practice is to have a “base” or “default” settings file. Put all the settings in here that MUST be present to tie all your Django apps together. Do NOT put security-related information in here. Do NOT put debug-related settings in here.

Because Django is python, you can go one step better in your file organization: Create a “settings” folder, with a “__init__.py” in it. Place all your various settings files here. Then, when you want to switch from the command-line, you can run this command:

$ python manage.py runserver --settings=settings.someconfig

Pitfall #2. Putting PRODUCTION settings in the default settings file.

You want to test your Django project with settings that will actually work on Production, right? Yes, or at least reasonably close. The temptation is to make your settings.py include all the Production settings, and then override them when you’re in development.

This is a great idea…until you get a new developer who checks out the code, then runs syncdb–and repopulates all your tables with the initial data fixtures…ON YOUR PRODUCTION DATABASE!

Rather, your base settings file should not contain any Production values. Instead, it should either contain values that will work for a local developer, or it should be missing these values entirely, so that the developer is forced to specify them.

Pitfall #3. Optionally including a “local” development settings file.

I’ve seen this in a number of projects. There is a “base_settings.py” file, which is good. But to allow the developer to specify alternate settings easily, I see a “settings.py” file that contains this code:

from base_settings import *
try:
    from local_settings import *
except:
    pass

There are two problems with this approach.

First, while this works when the local_settings.py file is valid, it FAILS SILENTLY when there is a bug in the local_settings.py file. And if your base_settings.py file contains security information for Production, you could accidentally be connected to a Production database while developing. (Can anyone say, “Wow! I just destroyed my production data!”)

The second problem with this approach is that it does not allow local_settings.py to override any of the values present in base_settings.py. Rather, local_settings.py can only REDEFINE these settings, because local_settings.py does not ever actually IMPORT them from base_settings.py In short, you can’t do this:

DATABASES['default']['NAME'] = 'my_dev_database'

If you try that, Python will complain than DATABASES is not defined. But…since the Exception is caught and ignored, you don’t know that it’s not working!

A Suggested Best-Practice for Django Settings Files

After falling into these pits multiple times, I’ve come up with this methodology for handling settings files. You’re welcome to use it, too. 🙂

1. Don’t store a settings.py file in your Source Code Control. This will force the local developer to either symlink (or copy on Windows) the relevant settings file to “settings.py”. Django will complain about a missing settings.py file–even if you use the “–settings” command-line option. Symlinking makes it super-easy to switch out one set of settings during development.
2. Separate out settings for base, production, QA, and local development. If you can, put them into a dedicated “settings” module, with __init__.py file. NOTE: Some server admins won’t want you storing production settings in your Source Code Control System. That’s OK. Just tell them where the base is, and have them override it in a settings file they keep somewhere else.
3. Import base settings into the other settings files. This allows you to override base settings, rather than having to redefine them.
4. BONUS: If you have to rely on system environment variables, domain names, or any other magic, then you should validate the values. Complain mightily about missing values!

Happy Django development! 🙂

Development server to the world!

This is a nifty trick you can use when developing a Django box. It’s straight from the Django reference manual @ https://docs.djangoproject.com/en/dev/ref/django-admin/

Note that the default IP address, 127.0.0.1, is not accessible from other machines on your network. To make your development server viewable to other machines on the network, use its own IP address (e.g. 192.168.2.1) or 0.0.0.0 or :: (with IPv6 enabled).

Develop Django on Linux, View on Windows with Putty Port-forwarding

I have an on-going project to integrate a Windows Desktop application (namely, QuickBooks) with web applications built on Django.There are two issues with this:

1. The QuickBooks Web Connector requires valid certificate HTTPS connections, except for testing–in which case you can use http://127.0.0.1:8000 (and other ports, presumably).
2. I prefer to develop on Linux.

To date, I have done most of my Django development on the Windows box itself. Django runs just fine on Windows. However, getting all the dependencies on Windows can be a real pain. I did find some pre-compiled binaries, so I was able to make this work. But it wasn’t really satisfactory, IMHO.

Putty to the rescue!

I tried this before, a while ago, but couldn’t get it working. Today, I figured it out. I’m sharing it, in case you run into the same situation.

Setup port-forwarding on Windows

In Putty, you have to add a port-forwarding link. Here’s what it looks like:

IMPORTANT: Don’t forget to click “Add”!

When you start this connection, you can view the Event Log to verify that the port-forwarding worked. (Thanks to http://the.earth.li/~sgtatham/putty/0.52/htmldoc/Chapter3.html#3.5)

Run Django on a network-accessible IP address

Typically, you run Django on http://127.0.0.1:8000. But that’s not accessible to the network. Instead, you should run the server on its network address:

$ python manage.py runserver 192.168.1.102:8000

Or, use this shortcut (thanks to CGD):

$ python manage.py runserver 0.0.0.0:8000

Connect!

Now, on your Windows box, you can browse to http://127.0.0.1:8000 and it will hit your Django server running on Linux.

Django ignoring model methods?!

So, today, I was trying to add a method to a Django model. Something like this:

class MyModel(models.Model):
    def shiny_new_method(self):
        pass

For some reason, no matter what I did, I could not see my shiny_new_method. I tried deleting all the *.pyc files, because I thought maybe Python or Django was keeping an old copy around. I tried changing it to inherit from (object) instead of from (models.Model). Nothing worked! It was driving me NUTS!

Tonight, I opened up the file again and searched for “MyModel”. Guess what? It was actually defined TWICE in the same file. And I was changing the second copy of it, which Django happily ignored (since it already had a definition of MyModel). At least…that’s what I think was happening.

Anyway, after I deleted the extra copy, I could see my shiny_new_method. Whew! It’s good to know I’m not crazy. (Well, that’s debatable.)

Environment variables in Python, Django, Windows and Hudson

I was having trouble with one of my Hudson builds. It was failing on Windows, but working fine on Ubuntu.

First, some background:

This build is a “free-style software project” written in Python, using the Django web framework. Python isn’t a compiled language, so there isn’t any real compilation going on. I just have Hudson doing a clean check-out (from Mercurial) and then running a prepared Python script.

I chose to use Hudson for this, so that I could verify that it worked on Ubuntu and Windows. Also, Hudson shows the history of builds, along with a nice trend graph that shows successes and failures. It’s handy to know that my “optimizations” actually do improve running time. (Like the latest, which cut the overall process down from 2.5 hours to 25 minutes.) Also, Hudson keeps the stdout from my Python script for each run, so I can do investigative log-searching after the fact, if I need to.

But, in short, Hudson gives me “fire and forget” for this long-running process. That means I can “hg push” and then go eat dinner, without babysitting the process.

The problem:

I made a change, then fired it off to the repository. Then, my Windows build started breaking. The one running on Ubuntu kept working just fine.

It’s a bit strange to explain, but here’s what was happening:

C:\path\to\hudson\jobs\thisproject\>python manage.py shell
>>> import settings
>>> settings.DATABASES['default']['NAME']
'/path/to/correct/db.sqlite3'
>>> from django.conf import settings as conf_settings
>>> conf_settings.DATABASES['default']['NAME']
'/path/to/wrong/oldversion/db.sqlite3'

I couldn’t figure out what was happening for a while. It seemed that “from django.conf import settings” was importing the wrong settings.py file!

Long story short, it turned out that my Windows PYTHONPATH environment variable was getting in the way. I had set it globally, using the “My Computer > Properties” dialog, so that every Command Prompt had that set. It was set to another version of the same project, at a different path.

So, I deleted that global PYTHONPATH setting. Then–and very important–I had to start a new Command Prompt. After that, it worked as expected.

Fixing Hudson

So, I kicked off a manual build via the Hudson Web interface. Surprise! It failed in exactly the same way as before. What gives?

Well, I was running Hudson via a Command Prompt invocation. (See it, now?) Yep, that Command Prompt shell had the old PYTHONPATH environment variable value. And every process that ran inside it also got that value passed to it. So, Hudson got it, then passed it off to Python.

I shut down Hudson, then restarted it. This time it worked.

Moral of the Story

Environment variables are very important. If you change them, you need to restart processes that are affected by them. This includes anything Python-related whenever you change the PYTHONPATH.

Why we love Django

We currently use (and love) Django, for these reasons:

• Very low learning curve.
• Good separation of concerns, using the Model-View-Controller (or, Model-Template-View) pattern
• Very clean templating language that keeps code out of the template
• Built-in ORM that supports PostgreSQL, MySQL and SQLite (with others available through 3rd parties)
• It builds your database table structure based on your python models
• Built-in Admin site that rivals phpMySQL (but works for all of the above databases)
• Almost never need to touch SQL
• Built-in unit tests of the core
• Easy to code unit and functional tests of your own stuff
• Built-in user permissions that are easily extensible
• Built-in user authentication
• Develop on your local box, deploy to a different OS or Database (if you’re careful)
• Free and open-source
• Great documentation and tutorials
• Works on Windows, Linux, Mac, etc.
• Uses python (“the world’s best programming language,” says Steve), which keeps it easy to read and understand
• amazing URLs (predictable, easy, and meaningful)
• Paul, who has experience with ASP, says the views are so much shorter than ASP code-behind

And that’s what we came up with in about 5 minutes.

Best thing you could do is download it and try the tutorial at:
http://www.djangoproject.com/

It would be the best two-weekend software project you’ll do for a long time. (Or your money back!)

Testing Django with Hudson CI

This article picks up where Setting up Hudson CI on Ubuntu left off.

Configuring Django Tests

In the last post, I almost had something working. But, I had forgotten to actually RUN the Django tests.

No worries. It was fairly trivial to setup this up. All I had to do was:

1. Click “Configure” from the project dashboard. (Be careful–there are several “configure” links in different places. I have yet to figure out what they all do.)
2. About half-way down the job configuration, select “Execute shell” from the “Add Build Step” dropdown.
3. Enter “python manage.py test”. (I wasn’t clear that this would work, but it seems to have.)
4. Click “Save”

After that, I re-ran the job. The “Console Output” now includes the Django test-runner output. Yay!

Making Django produce XML test output

At this point, I’m going to “borrow” from this nice write-ups of
Continuous Integration with Django and Hudson CI (Day 1) (for Django 1.1) and http://www.pointlessrants.com/2010/12/unittest-xml-reporting-multi-database-django-1-2/ (for Django 1.2+).

Well, I don’t really like having to install extra stuff in my project, or on my server, just to get the XML output. However, I *do* want to see breakage reports in Hudson. It seems like my options are: sptest (an apparently unsupported Ubuntu package); nosetest (which has always turned me off, for some reason); and unittest-xml-reporting (with built-in support for Django). Let’s install and use the latter, but manually (not using “pip”, for my own reasons):

If you’re using Django 1.1, grab this tarball:

$ wget https://github.com/danielfm/unittest-xml-reporting/tarball/master --no-check-certificate
$ tar xvf master
$ cd danielfm-unittest-xml-reporting-8972b5f/
$ sudo python setup.py install

If you’re on Django 1.2+, grab this one (Oy, open source!):

$ wget https://github.com/daspecster/unittest-xml-reporting/tarball/master --no-check-certificate
$ tar xvf master
$ cd daspecster-unittest-xml-reporting-a573346/
$ sudo python setup.py install

I always like to verify that it ran:

$ python
>>> import xmlrunner

Now, I created a separate hudson_settings.py file, so that I don’t have to use the XML output unless I want to. Here’s what the file contains:

from settings import *
#For Django 1.1:
#TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.run_tests'
#For Django 1.2+:
TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner'
TEST_OUTPUT_VERBOSE = True
TEST_OUTPUT_DESCRIPTIONS = True
TEST_OUTPUT_DIR = 'xmlrunner'

Don’t forget to add and check-in hudson_settings.py!

Now, back in the Hudson project, I need to configure the command-line Build Action that Hudson has been using, like this:

python manage.py test --settings=hudson_settings

Also, I have to tell Hudson to grab the XML results:

• Check “Publish JUnit test result report”
• Enter the path to the XML files. For this project, it is simply “xmlrunner/*.xml”. (NOTE: I got a big red warning about “‘xmlrunner/*.xml’ doesn’t match anything: even ‘xmlrunner’ doesn’t exist”. That’s because we haven’t run the XML tests yet! Keep going.)

OK. Let’s see if that works…COOL! It does! 😀

Now, after my build runs, I get some test results by “package”–aka, Django app.

Bonus: Automatic testing

The django-dogwood project is hosted on Google Code, which supports Post-commit Web Hooks. I imagine that these can be used to trigger a build in Hudson. However, my Hudson server is my localbox, so I have no URL to give to Google Code, for a web-hook.

What I can do is schedule a periodic build. That is, now that I found the help text about the cron-like syntax. 😉 I figure, building once an hour, between 6 AM and 11 PM should be good enough for this project.