Improve Django shell script speed with manual transaction management

I have a long-running Django process that takes too long to run via a view. Either the browser would time out or my patience would run out, without having any feedback about what’s happening.

So, I run it via Django’s interactive shell. That way, I at least can see the logging messages as they go by.

I used hotshot to profile my code, and found that the _commit() function was taking the most time. So, the obvious thought is:

How do I turn off auto-commit, and manually control the transactions?

Well, for views, you can use the TransactionMiddleware decorator. It works fine…for view functions. But when you try it on non-view functions, it is totally irrelevant.

After much searching, I found this excellent post about using SQLite PRAGMA directives (which I have yet to try). The comments pointed me to the simpler answer I was seeking: how to manually control transactions in non-view functions.

The post by multani was helpful, but he omitted one important detail: You need to call transaction.leave_transaction_management() when you’re done.

That chopped about 80% off of the execution time for my particular method. 🙂

Profiling Django with hotshot

I found this article talking about Profiling Django. So, I read up on hotshot. The standard documentation has an example, which actually works. (Although, I did have to install the python-profiler package on Ubuntu 10.04.)

I tried to copy-and-paste the example, and modify it just enough to profile my own function. I found that this line did not work:

benchtime, stones = prof.runcall(myfunc)

It gave me a python error about “Too many values to unpack” from a None value. I scratched my head and wondered,

“Why is prof.runcall() not returning a tuple of (benchtime, stones)?”

After grokking the source code for runcall(), I found that runcall() actually returns the results of whatever function it’s profiling. Aha!My method wasn’t returning anything; hence, there was nothing to unpack. Here’s the way I fixed the line:

prof.runcall(myfunc)

And that worked beautifully. 🙂

PHP to Django Converter

Right now, it’s just a tiny little idea at http://code.google.com/p/php2django/. The project ethos is embodied here:

PHP has a lot of well-built code. Django is relatively new, but is gaining speed. Rather than reinvent the wheel, why not borrow code ideas from good working PHP solutions?

Django SAML 2.0 Identity Provider

New project! http://code.google.com/p/django-saml2-idp/

The Security Assertion Markup Language (SAML) 2.0 is useful for helping organizations work together. By establishing a trust relationship among their systems, their users can log into each others’ systems easily. (Ready to fill up your brain? Read more about the SAML 2.0 standard.)

Using the helpful starter code, the Google code project is now able to generate signed assertions. (As of source code revision 5.)

The first goal (version 0.1) is to implement the Web Browser SSO (SingleSignOn) Profile using the HTTP POST binding. (After that, maybe I’ll add more bindings and profiles.)

If you know SAML 2.0, I’d love to have your help!

Why use Python and Django?

DISCLAIMER: This could also be called, “Why not use Java and JCR?” But I’m going to “pick on” Java and JCR, simply because the example code was right there and easy to grab. I mean no ill feelings towards Java, the JCR or anyone who uses these. This post is simply a comparison, so that you can see why I emphasize Python and Django for web development.

OK, onward!

I was reading about Jackrabbit today, simply because I like to know what’s out there.  I found this example, for how to create a “PressRelease” object using Jackrabbit:

package org.apache.jackrabbit.ocm.model;

import java.util.Date;

import org.apache.jackrabbit.ocm.mapper.impl.annotation.Field;
import org.apache.jackrabbit.ocm.mapper.impl.annotation.Node;

@Node
public class PressRelease
{
	@Field(path=true) String path;
	@Field String title;
	@Field Date pubDate;
	@Field String content;

	public String getPath() {
		return path;
	}
	public void setPath(String path) {
		this.path = path;
	}
	public String getContent() {
		return content;
	}
	public void setContent(String content) {
		this.content = content;
	}
	public Date getPubDate() {
		return pubDate;
	}
	public void setPubDate(Date pubDate) {
		this.pubDate = pubDate;
	}
	public String getTitle() {
		return title;
	}
	public void setTitle(String title) {
		this.title = title;
	}

}

That’s 41 lines. Now, you could do something very similar in 9 lines with Django, like this:

from django import models

IDUNNO = 255 #maybe, I'm guessing

class PressRelease(models.Model):
    path = models.CharField(max_length=IDUNNO)
    title = models.CharField(max_length=IDUNNO)
    pub_date = models.DateTimeField(auto_add=True)
    content = models.TextField()

Now, let’s actually do something with this model. Like create an object and save it. In Jackrabbit, you have to do this: (NOTE: I’ve combined two code snippets, because I think that’s what I would have to do to actually make it work.)

// In order to save a PressRelease object, you have to instantiate an Object Content Manager component
List classes = new ArrayList();	
classes.add(PressRelease.class); // Call this method for each persistent class
		
Mapper mapper = new AnnotationMapperImpl(classes);
ObjectContentManager ocm =  new ObjectContentManagerImpl(session, mapper);	

// Insert an object
System.out.println("Insert a press release in the repository");
PressRelease pressRelease = new PressRelease();
pressRelease.setPath("/newtutorial");
pressRelease.setTitle("This is the first tutorial on OCM");
pressRelease.setPubDate(new Date());
pressRelease.setContent("Many Jackrabbit users ask to the dev team to make a tutorial on OCM");
			
ocm.insert(pressRelease);
ocm.save();
			
// Retrieve 
System.out.println("Retrieve a press release from the repository");
pressRelease = (PressRelease) ocm.getObject("/newtutorial");
System.out.println("PressRelease title : " + pressRelease.getTitle());
			
// Delete
System.out.println("Remove a press release from the repository");
ocm.remove(pressRelease);
ocm.save();

(26 Lines?)

Now, let’s try that in Django:

from some_app.models import PressRelease

// Insert an object
print "Insert a press release in the repository"
pressRelease = PressRelease()
pressRelease.path = "/newtutorial"
pressRelease.title = "This is the first tutorial on OCM"
pressRelease.content = "Many Jackrabbit users ask to the dev team to make a tutorial on OCM"
pressRelease.save()
	
// Retrieve 
print "Retrieve a press release from the repository"
pressRelease = PressRelease.objects.get(path="/newtutorial")
print "PressRelease title : " + pressRelease.title
			
// Delete
print "Remove a press release from the repository"
pressRelease.delete()

Now, that’s 18 lines, so it’s not so much difference on the line count only. But look at the readability.

Can you see why I really, really like Django? 🙂

Top-level URL manipulation in Django

This is a top-level example of how you could hack the URLs of your site at the highest level, without having to mess with any code at lower levels. Of course, this presumes that you know exactly how the URLs work for each app and are confident in altering them. It also assumes that templates use the {% url %} template tag, and that calls to redirect() use the reverse() function based on view names.

Consider this alt_urls.py file. To activate it, you would need to specify ROOT_URLCONF=’alt_urls’ in settings.py.

First, some setup:

#Standard import for urls.py files:
from django.conf.urls.defaults import *

# Grab everything from the original URLs:
from urls import *

# Copy the 'urlpatterns' module property, so we can create an altered one.
old_urlpatterns = urlpatterns

Adding New Patterns

Example of how you might create an altered ‘urlpatterns’ that adds some new URLs for your system. NOTE: This relies on first-in-wins, which only works if the URLs you define here are different from the ones the apps define. (See OVERRIDING below for more details.)

urlpatterns = patterns('',
    # ...like, injecting a pattern ahead of another, to short-circuit things,
    #    or trigger middleware, maybe.
    (r'^altered/', include('urls')),

    # ...or creating a special URL to a specific view on an object (crazy!):
    #NOTE: We add these as named URLs, but they are names that the apps don't
    #      know about. So, if you view "/narnia/" then click "edit", but
    #      "cancel," it will not take you back to "/narnia". Rather it will
    #      take you back to "/pageview/4" (because the app looks up the view
    #      by the 'view_page' name (which is overridden in the next section).
    url(r'^narnia/$', 'pages.views.view_page', {'pk': 2}, name='narnia_home'),
    url(r'^calormen/$', 'pages.views.view_page', {'pk': 4}, name='calormen_home'),
)
# And, don't forget to append the old patterns! :)
urlpatterns += old_urlpatterns

Overriding Existing Patterns

Example of how you could modify named patterns. NOTE: This seems very tricky, because it relies on last-in-wins, which is not documented anywhere that I can find. (But a better way is coming in the next example, TWEAKING.)

urlpatterns += patterns('',
    # ...or shortening URLs to a specific app:
    (r'^pages/', include('pages.urls')),

    # ...or shortening a URL to a specific view:
    #    (This requires knowledge of the view name, defined in the app URLs.)
    url(r'^pageview/(\d+)', 'pages.views.view_page', name='view_page'),
)

Tweaking Individual Patterns

It’s fairly easy to replace individual named patterns, like this:

tweaked_patterns = []
for x in urlpatterns:
    try:
         if x.name == 'calormen_home':
             tweaked_patterns += patterns('',
                 url(r'^CALORMEN/$', 'pages.views.view_page', {'pk': 4}, name='calormen_home'),
             )
             continue
    except:
        pass #ignore exceptions...
        #... and just keep the pattern
        tweaked_patterns.append(x)
urlpatterns = tweaked_patterns

CakePHP vs Django

I’ve been curious about CakePHP, since a friend has a site built with that. I’ve been tossing and turning at night, wondering whether Django is the right framework to continue pursuing.

Finally! a working Django non-root URL with mod_wsgi

It’s pretty simple.  I want to serve Django apps from a non-root URL on my webserver.  I’ve been fighting with this for a long time and finally–FINALLY!–figured it out. 

django-opensocial

This project has been around for several years, with no activity.  The project owner (“jtauber”) just granted me co-owner status, so I can begin contributing code.  Thanks!

The goal is to make a baseline OpenSocial container in Django, including at least one baseline OpenSocial widget also in Django.  This baseline project will be the foundation on which missions geeks can build, to enable their systems for OpenSocial integration.

See that project page for more details, as they come.

open-scriptures

I was totally blown away when I found this Google Code project:

http://code.google.com/p/open-scriptures/

It’s in Django, no less!