Speed up PostgreSQL with Django tests

My tests were taking forever. OK, only about 251 seconds (4 minutes), but that’s forever when you only have 11 tests! (In this one app.)

A lot of people talk about how to speed up PostgreSQL, but here it is in a nutshell:

Turn off fsync and full_page_writes in postgresql.conf

Specifically, here’s how I did it on my Linx Mint 13 box:

1. Edit postgresql.conf.

$ sudo vim /etc/postgresql/9.1/main/postgresql.conf

Change these lines and saved:

fsync = off
full_page_writes = off

2. Restart postgresql.

$ sudo service postgresql restart

Now, when I run my 11 tests, it only takes 34 seconds. That’s 8x faster!

NOTE: Probably don’t do this on your Production box.

Secure bcfg2 web reports with apache2 HTTP authentication

The problem

bcfg2 web reports doesn’t require any authentication out-of-the-box. (As of today.) This means that anyone who knows the URL of your bcfg2 web reports can see (and manipulate?) your server and/or clients.

Possible solutions that I didn’t attempt

The settings.py file contains an AUTHORIZED_GROUP setting. It looks like this activates the NISAuth authentication backend. I have no idea what NIS is, so I’m moving on.

The settings.py also includes the standard Django authentication backend, so theoretically, you could hack the views.py and use the @login_required decorator. But I’m feeling lazy today and want an easier solution.

The simple solution: Apache2 HTTP Authentication

I only care about one user: myself. So, I created a password file, using this command:

 sudo htpasswd -c /etc/apache2/bcfg2-passwords myusername

Then, I added this block of code in /etc/apach2/conf.d/bcfg2.conf:

    <Location /bcfg2>
        AuthType Basic
        AuthName "Bcfg2 Web Reports"
        AuthBasicProvider file
        AuthUserFile /etc/apache2/bcgf2-passwords
        Require user myusername
    </Location>

Disclosure

I did this with:

• Ubuntu 12.04.2 LTS, Precise Pangolin
• 64-bit architecture
• on Rackspace Cloud
• Bcfg2 Version 1.2.2
• HTTPS enabled on my server (don’t forget to do this or your password will be passed in cleartext!)

I also followed these instructions to install bcfg2-web (after installing bcfg2):

http://docs.bcfg2.org/appendix/guides/web-reports-install.html#appendix-guides-web-reports-install

Thanks to http://stackoverflow.com/questions/8417810/deploying-a-django-app-on-apache-mod-wsgi-with-http-auth

Also, thanks to solj and scofflaw on the #bcfg2 IRC channel.

whatis man

Linux just got shinier for me! I love its command-line tools, and just stumbled upon one that’s new to me. It’s the whatis command. Basically, it tells you whatever the system knows about any type of keyword.

So, try:
john@marcosmurf:~$ whatis whatis
whatis (1) - display manual page descriptions

I already know that *man* is the manual database. So, just for fun, I ran:

john@marcosmurf:~$ whatis man
man (7) - macros to format man pages
man (1) - an interface to the on-line reference manuals

And, back to the age-old question:

john@marcosmurf:~$ whatis woman
woman: nothing appropriate.


Reverting an “oops” with Subversion

Last week I was in the situation of needing to revert a series of changes I’d made to a project in SVN. This was a Python script working with a GIS API. I’d discovered an API function I hadn’t noticed before, one that seemed to kill several birds with one stone, and make the code simpler. Score! But after a week, and several commits, the new approach turned out to be more of a headache than a benefit. So how do you revert changes in SVN?

Ideally, you made your risky changes on a branch; and if they don’t pan out, you just don’t merge the branch into the trunk. But what if you were committing to the trunk all along?

It’s easy enough to roll back to the last “known good” revision: you just update to that revision.  (My last known good revision was 12634, for the sake of discussion.) In TortoiseSVN, you use the menu item “Update to revision…” But that doesn’t quite get us there. In order to continue working, we have to make r12634 the HEAD. Otherwise, if we try to commit new changes, we’ll be told we can’t commit until we update. And when we update, we’ll get all those changes back that we’re trying to get rid of.

The answer is out there, for a little searching. I learned from posts like this that in order to set the HEAD revision back to an older state, you need to reverse-merge the unwanted changes onto the trunk, and then commit.

First, I created a branch to hold the unwanted changes, so I could find them easily in the future. (I knew I would want to go through these changes later and glean a few nuggets that were worth preserving.) The last revision was 12662, so I created a branch from that revision.

For the reverse-merge, I used the svn command-line client. My colleague Steve told me that TSVN’s Merge dialog required you to choose from a set of confusing, counter-intuitive options for merge type. Since I find SVN merging confusing enough already, I decided to sidestep one level of obfuscation, and go with the more widely documented / peer-reviewed / tested mechanisms of the command-line. Here’s the reverse-merge… the same as a normal merge, but swap the order of the revision numbers:

svn merge -r12662:12634 . 

But do make sure that the first number there (before the colon) really is the current HEAD. Otherwise, the reverse merge will likely lead to conflicts. And we can all do with less conflict.

Once that’s done, the reverse-merge is just another set of changes to commit. The fact that these new changes happen to correspond to a past revision, r12634, is not something SVN pays attention to (why should it?). So now we just commit the changes: r12663. Now the HEAD revision on the trunk  (12663) consists of code that is the same as the code at r12634.

Re-merging lost nuggets

The next phase was going manually through the changes that I’d just reverted, and picking out a few that were still valid and useful.

My favorite tool for interactive, computer-assisted 2-way merging is WinMerge. (This tool has been such a staple in my toolbox–or  even a screwdriver in my toolbox–that I have donated to the project. I think. If I haven’t, I would!)

Thankfully, this could be done as a 2-way merge, or a series of them anyway. Since the code I was applying changes to, the HEAD revision, was the same code as the common ancestor (r12634), I could simply compare the working copy of the file with each of its revisions between 12634 and 12662, manually merging as I went along.

I used TSVN’s Show Log menu option to list the revisions to the file (there were a lot less than 62-34, thankfully!). I selected each revision since 12634 in the upper pane, then right-clicked on the relevant file in the lower pane and clicked “Save revision to…” This offered by default to save the filename with a revision number tacked on, e.g. foo-12635.py, which was perfect for the purpose.

That gave me a solid basis to start using WinMerge on, first merging the working copy version (foo.py) with the earliest revision (foo-12635.py), interactively choosing one or two changes to bring into the working copy. After saving the result to foo.py, I could then merge the modified foo.py with foo-12637.py, etc. Once all this re-merging was done, I was able to commit again, and feel confident that I hadn’t lost anything important. Obviously, after all that manual merging I still had some testing to do, but the hard part was done.

(To do: make a little diagram of what our SVN revision tree looked like, before and after.)

Lesson learned?

Why I wasn’t using a branch in the first place, for these changes that turned out to be problematic? Would I do it differently next time?

It’s hard to say. Hindsight is great, but at the time, the change I was making didn’t seem complicated. You can’t create a new branch for every little change, and this one seemed as routine as any other. Besides, I was the only developer on the project at the time, and only one file was involved. It was only as I got deeper into testing my changes, and discovering the extent of the implications, that it became apparent how much the code would actually need to change in order to round out the new approach.

Do you have a rule of thumb for when to create a new branch? I’ve been using JIRA issues to track tasks on this project. Maybe creating a new branch for each JIRA issue is a good idea? I’ve never really gotten used to merging branches back into the trunk, but maybe it’s time I learned to make that a regular practice.

How to install the Campfire plugin for Pidgin on Linux

WARNING: This software is untested and may not work.

If you’re still reading, then you take responsibility for what the software may do to your computer. (In other words, I probably can’t help you fix anything that blows up.)

If you’re still reading, then follow these instructions (for Linux Mint; other flavors of linux or distros…YMMV):

1. If you don’t already have them, install the standard compile and make packages for linux. (I forget what they are. Look them up.)
2. Install the development headers for libpurple:  sudo apt-get install libpurple-dev
3. Download the source code for the plugin here: https://github.com/jrfoell/campfire-libpurple
   (I recommend grabbing the ZIP file and just extracting the files from there. But if you have git, then feel free to use it instead.)
4. In the folder where the MAKEFILE lives, run these commands:
   sudo make
   sudo make install
5. Restart Pidgin.

Now that it installed, here’s how to configure it for your account:

1. Open “Manage Accounts”.
2. Click “New…”
3. Select the “Campfire” protocol.
4. Enter your username and the hostname of your campfire server.
5. Get an API key from the “my info” link on your campfire server’s web interface. (In short: use a web browser to get this info.)
6. Enter the API key in the “Advanced” tab.
7. Save the account.

Theoretically, you should now be able to chat on Campfire using Pidgin. Here’s how to join a room:

1. In the Pidgin Buddy List, select menu item Buddies > “Join a chat…”
2. Select the “Campfire” protocol from the dropdown list.
3. Click “Room List” to get a list of rooms available on your Campfire server.
4. Select the room you wish to join, then click “Join Room”. A new chat window should open up to that room.
5. You may have to close the “Join a chat” dialog box.

sshuttle DNS stopped working…and how i fixed it

I really like sshuttle. I use it to connect my Linux box to a client’s VPN through the Macbook they have provided for this project. (They offered a Windows machine, but I doubt this would even be possible with Microsoft’s poor POSIX support.)

Anyway, I work from multiple locations: home, office, coffeeshop, etc. And these different locations have different setups. On my home LAN and at the office, I’ve got dedicated wires for each machine. But at the coffeeshop and other places, I use a crossover cable for SSH between the machines and use the Wireless cards on each box to connect to the Internet. This can be tricky with sshuttle.

Today, I plugged into a different office network and just could not get sshuttle to work. It would connect, but I could not hit any IP addresses or DNS names on that connection. It didn’t seem to matter what options I tried with sshuttle.

The clue came when I ran “nslookup google.com” on the client box–WITHOUT connecting to sshuttle. It told me it was using “127.0.0.1″ as the DNS server. But wait…doesn’t sshuttle also use 127.0.0.1 to serve its own DNS records? Ah…yes, it does.

Yesterday, after leaving the coffeeshop, I blew away my /etc/resolv.conf, because it seemed unnecessary. After all, Linux Mint doesn’t use that file, so what’s the harm?  Well, for reasons I don’t understand, without /etc/resolv.conf, my linux box fell back to using 127.0.0.1 as its DNS server. And this prevented sshuttle from providing DNS entries from the server’s network.

I found out the IP Address for the nameserver here (by running nslookup on another box) and added this line to /etc/resolv.conf:

nameserver 209.18.47.61

Once I did that, sshuttle DNS worked again beautifully.

If this long ramble helps you, please comment and let me know. Thanks!

Screen FTW

If you use ssh on *nix servers, you owe it to yourself to learn and use screen. (It takes about 15 minutes to Google everything you’ll need to know to use it.)

My .screenrc file does three things for me:

1. It automatically “detaches” when my connection drops. This lets me reconnect to whatever I had going with “screen -R” when I next log in.
2. It processes each screen as a login shell, so that all my settings, aliases, etc. are available in each screen.
3. It shows me a nicely formatted hardstatus line at the bottom (which I got from here).

Here’s the actual file:

autodetach on
defshell -bash

# Window list at the bottom. hostname, centered tabs and redmarked active windows:
hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'

In case the formatting in that last line gets cut off in your browser, here it is:
%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]