Environment variables in Python, Django, Windows and Hudson

I was having trouble with one of my Hudson builds. It was failing on Windows, but working fine on Ubuntu.

First, some background:

This build is a “free-style software project” written in Python, using the Django web framework. Python isn’t a compiled language, so there isn’t any real compilation going on. I just have Hudson doing a clean check-out (from Mercurial) and then running a prepared Python script.

I chose to use Hudson for this, so that I could verify that it worked on Ubuntu and Windows. Also, Hudson shows the history of builds, along with a nice trend graph that shows successes and failures. It’s handy to know that my “optimizations” actually do improve running time. (Like the latest, which cut the overall process down from 2.5 hours to 25 minutes.) Also, Hudson keeps the stdout from my Python script for each run, so I can do investigative log-searching after the fact, if I need to.

But, in short, Hudson gives me “fire and forget” for this long-running process. That means I can “hg push” and then go eat dinner, without babysitting the process.

The problem:

I made a change, then fired it off to the repository. Then, my Windows build started breaking. The one running on Ubuntu kept working just fine.

It’s a bit strange to explain, but here’s what was happening:

C:\path\to\hudson\jobs\thisproject\>python manage.py shell
>>> import settings
>>> settings.DATABASES['default']['NAME']
'/path/to/correct/db.sqlite3'
>>> from django.conf import settings as conf_settings
>>> conf_settings.DATABASES['default']['NAME']
'/path/to/wrong/oldversion/db.sqlite3'

I couldn’t figure out what was happening for a while. It seemed that “from django.conf import settings” was importing the wrong settings.py file!

Long story short, it turned out that my Windows PYTHONPATH environment variable was getting in the way. I had set it globally, using the “My Computer > Properties” dialog, so that every Command Prompt had that set. It was set to another version of the same project, at a different path.

So, I deleted that global PYTHONPATH setting. Then–and very important–I had to start a new Command Prompt. After that, it worked as expected.

Fixing Hudson

So, I kicked off a manual build via the Hudson Web interface. Surprise! It failed in exactly the same way as before. What gives?

Well, I was running Hudson via a Command Prompt invocation. (See it, now?) Yep, that Command Prompt shell had the old PYTHONPATH environment variable value. And every process that ran inside it also got that value passed to it. So, Hudson got it, then passed it off to Python.

I shut down Hudson, then restarted it. This time it worked.

Moral of the Story

Environment variables are very important. If you change them, you need to restart processes that are affected by them. This includes anything Python-related whenever you change the PYTHONPATH.