The other side of the process is taking a search string, breaking it down into words and finding all of the posts that match the terms. I even have my own PageRank algorithm:

@srch.route('/', methods=['GET', 'POST'])
def search():
    if request.method == 'POST':
        terms = get_words(request.form['search_string'])
    else:
        terms = get_words(request.args.get('s', ''))
    results = []
    for t in terms:
        indexFilename = "%s/%s" % (app.config['SEARCH_INDEX_DIR'], t)
        if os.path.exists(indexFilename):
            indexFile = codecs.open(indexFilename, encoding='utf-8', mode='r')
            results = indexFile.read().splitlines() + results
            indexFile.close()

    result_set = set(results)

    ranked_results = []
    for r in result_set:
        ranked_results.append((results.count(r), r, os.path.split(r)[1]))

    ranked_results.sort()
    ranked_results.reverse()

    return render_template('%s/search_results.html' % app.config['SITE_TEMPLATE'],
                       terms=" ".join(terms),
                       results=ranked_results)

This is the only part of my website that isn't served as static HTML. I guess in theory I could create a search results page for every possible word combination, but that would be stupid.

I ran into a problem though. The main application runs on a special subdomain behind a login, separate to the main site you're reading now. The search part though is accessed anonymously and from the main site domain.

I wasn't sure at first how to do this with Flask - I knew that I wanted to keep everything all neatly within the same package though, as it all belongs together.

My inexperience with Python lead me to creating a .wsgi file in the static site that only imported the pydimension.search module. I thought that would be quite clever, however, it still imports the whole of the Flask app, so I just ended up with a new entrypoint to the admin application.

After briefly giving up, I realised that the app object in my package was really just an arbitrary instantiation of the Flask class. I could create a new one and just add the /search route to it.

I modified the package __init__.py file:

from flask import Flask

app = Flask(__name__)
app.config.from_pyfile('site_settings.cfg')

srch = Flask(__name__)
srch.config.from_pyfile('site_settings.cfg')

import pyDimension.system
import pyDimension.access_control
import pyDimension.views
import pyDimension.search

then I changed the decorator for the search() function as you can see above. Initially, I routed it like this: @srch.route('/search', methods=['GET', 'POST']) and started my WSGI Daemon at the root of the domain, but I found that caused mod_wsgi to handle all request to the domain and just return 404 errors for everything but /search, instead, when I configured Apache, I started the daemon at /search:

<VirtualHost *:80>
    ServerName flasktest.subdimension.co.uk

    WSGIDaemonProcess flaskTest user=flask group=www-data threads=5 home=/<redacted>/searchTest
    WSGIScriptAlias /search /<redacted>/flaskTest/searchTest.wsgi

    <Directory /<redacted>/searchTest>
        WSGIProcessGroup searchTest
        WSGIApplicationGroup %{GLOBAL}
        WSGIScriptReloading On
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

In the .wsgi file for the static site domain, I imported srch from pyDimension instead of app:

import sys

sys.path.append('/redacted/pyDimension')

from pyDimension import srch as application

It worked!

PUNCTUATION = "#$%&"'()*+,./:;<=>!?@[]^`{|}~"

def get_words(text):
    depunc = text.lower()
    for p in PUNCTUATION:
        depunc = depunc.replace(p, ' ')

and it worked fine on all 3 machines that I use variously throughout the day, one OS X, one Windows 7 and one Ubuntu. Yesterday I deployed to Apache for the next stage of testing and the whole thing fell over!

Flask isn't much help when it isn't running through its development server so it took me a lot of poking around to work out what was going on. Finally, I worked out that it was falling over at the post that started all of this off. The quote I had copied and pasted:

If you only know PHP and you’re curious to learn something else, give the Python tutorial a whirl and try Flask for the web stuff.

it has a 'smart quote' in it for the word you're. I'm lazy and I've never bothered to use them myself, but lots of people use them and they do pop up in a couple more of my posts where I've copied quotes. The Flask development server didn't care about them at all, it happily created index files with these characters in the name - I had noticed but decided I didn't care just yet.

Once I deployed through mod_wsgi, this didn't work any more. I have no idea why and I'm not sure where to start finding out either. Instead, I modified my get_words() function:

depunc = re.sub('[^a-z_\-\s]', ' ', text.lower())

It probably even runs faster since it's using the regex module (written in C I believe), rather than iterating around characters in a string. I did think about adding the various other characters to the PUNCTUATION constant, but since they're Unicode characters, I knew I'd never get them all, so I just filter out everything that is not recognisable as a word that I have written. I will revisit this one day, if I ever find myself writing Arabic or French!

I'm still working on my Pythonic Static Blogging system, dubbed pyDimension for want of inspiration, and just finished implementing my static search indexing system. Python didn't come with an obvious way to generate a list of word from a body of text like PHP does, so I had to implement my own.

In PHP, I generated a list of words to use as indexes with this:

$words = array_unique(
    str_word_count(
        strtolower(
            str_replace("'", '', 
                strip_tags($text)))
    , 1));

When you pass the 1 parameter to PHP's str_word_count() it returns an array of words and it's quite clever about what it considers a word.

I couldn't find anything obvious in Python's Standard Library, so with a little bit of tinkering, I came up with this:

PUNCTUATION = "#$%&"'()*+,./:;<=>!?@[]^`{|}~"

def get_words(text):
    depunc = text.lower()
    for p in PUNCTUATION:
        depunc = depunc.replace(p, ' ')

    all_words = set(depunc.split())
    words = []

    for w in all_words:
        if len(w) > 2 and not w.isdigit():
            words.append(w.strip('-'))

    return words

The string package has a punctuation constant, but I wanted to keep underscores and hyphens, that way get_words would be indexed as the full method name, rather than 'get' and 'words', which is less meaningful. I also decided to exclude words of one or two characters, and numbers. Ideally I wanted to keep numbers, as they are useful - for example, in my post about Dropbox LanSync, indexing the port number would be useful. However, because of the way I remove punctuation, numbers with significant formatting, such as dates or times wouldn't be indexed properly, so I decided to leave them out.

Once I have a list of all the words in a post or page, I add the URL of that item to a list stored in a file named for each word.

Update

This doesn't work when it's running through mod_wsgi so I changed it

My first instinct was to panic: "OMG, I've been hacked and I'm spewing spam from my server, I should never have thought I was ever capable of running these things myself".

This was shortly replaced with "Hmm, that port number looks oddly 'reasonable', it must be something 'friendly'." I took a shot in the dark and Googled "Dropbox port 17500". Hits! Phew!

Dropbox has a nifty feature called LANSync, where it will try to find other machines on your local network and sync with them. It's a clever idea that means you're not forever sending the same files up and down the internet when you use it on multiple machines (and after all, what's the point of Dropbox if not to keep multiple machines in sync!).

I knew about this and had naively thought that it would just get ignored in a VPS environment - I'm a bit ignorant of how it all works really.

Apparently not though. My server has been spamming my VPS neighbours every 30 seconds for the last few weeks. Whoops!

After looking through the documentation I linked in my post about setting this all up in the first place, I noticed a distinct lack of any help with switching off LANSync from the command line - the links they provide are all broken now (nice going!).

Luckily I'd taken the time to install the Dropbox CLI, so I ran it:

$  ./bin/dropbox.py

Dropbox command-line interface

commands:

Note: use dropbox help <command> to view usage for a specific command.

 status       get current status of the dropboxd
 help         provide help
 puburl       get public url of a file in your dropbox
 stop         stop dropboxd
 running      return whether dropbox is running
 start        start dropboxd
 filestatus   get current sync status of one or more files
 ls           list directory contents with current sync status
 autostart    automatically start dropbox at login
 exclude      ignores/excludes a directory from syncing
 lansync      enables or disables LAN sync

Ah hah! Last one looks like the ticket:

$  ./bin/dropbox.py lansync

enables or disables LAN sync
dropbox lansync [y/n]

options:
  y  dropbox will use LAN sync (default)
  n  dropbox will not use LAN sync

$  ./bin/dropbox.py lansync n

hmm, no feedback, always helpful!

I've never really been a belt-and-braces kind of person, but I don't want to get shut down by my VPS provider, so a bit more research turned up iptables which is installed on all Ubuntu releases.

$  /sbin/iptables -A OUTPUT -p udp --dport 17500 -j DROP
$  iptables-save

which should block it from getting out if it ever starts up again. Sorted.

if '/' in sPath:
    ...
    month = [sPath.split('/')[1] for x in range(len(days))]

sPath was built by walking down into a directory and I used it to work out if I was looking at directories for days or months. This little fragment of code worked fine, until today, when I was running it on my Windows machine.

Windows uses the \ character to separate directories, Python knows this and it helpfully provides a set of functions that deal with it automatically. I just didn't use them:

if os.path.split(sPath)[0] != '':
    ...
    month = [os.path.split(sPath)[1] for x in range(len(days))]

os.path knows all about directory separators.

Out of the box, the flask framework comes with its own development server, so far I've only used this but eventually I would be deploying it onto my main server running apache. I decided I needed to make sure I could get this working without too much trouble so I made a simple Hello World application along the lines of the main flask tutorial.

I uploaded it to a folder on my server and SSH'd in to start configuring things. I'd already set up a sub domain to use and pointed it at the server too.

I found that the flask documentation for setting up with Apache made the assumption that you'd already worked with WSGI applications and just needed the flask way so it took a bit of digging and re-reading.

First, I created a new shell-less user to run my app as. I could have just used the Apache user (www-data), but I wanted to be able to distinguish files that had been molested by my flask app:

create a new user without a home directory:

$  useradd -M flask

remove shell:

$  usermod -s /bin/false flask

Finally, lock the account to prevent logging in:

$  usermod -L flask

I also added the user to the Apache www-data group. This makes it easier to work with file permissions later as I can keep ownership of them with my own user, but allow the www-data group access.

$  adduser flask www-data

Next I created flaskTest.wsgi in my application's folder:

import sys

sys.path.append('/<redacted>/flaskTest')

from flaskTest import app as application

Initially I didn't think I needed to import the path to the app as I assumed it would be started in its own directory, however, I realised later that mod_wsgi starts apps with the same base directory a Apache, which in my case was the system root.

Finally I created a new Apache VirtualHost config in /etc/apache2/sites-available:

<VirtualHost *:80>
    ServerName flasktest.subdimension.co.uk

    WSGIDaemonProcess flaskTest user=flask group=www-data threads=5 home=/<redacted>/flaskTest
    WSGIScriptAlias / /<redacted>/flaskTest/flaskTest.wsgi

    <Directory /<redacted>/flaskTest>
        WSGIProcessGroup flaskTest
        WSGIApplicationGroup %{GLOBAL}
        WSGIScriptReloading On
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

Adding the WSGIScriptReloading directive meant that any time I made changes to my application files, I could simply $  touch flaskTest.wsgi and the app would restart with changes applied. I even found a python-based implementation of touch that I could use if I wanted to be able to restart the app from within the app!

As mentioned above, by default, the application starts as if it were called from the server root. The WSGI Documentation gave me the extra home option for the WSGIDaemonProcess directive, which changes that to wherever you want it.

I had to make one final change to the configuration file from the documented example, which was to specify the port for the VirtualHost *:80. Without this it over-rode my other virtual hosts and everything ended up pointing to the same place!

With all this done, enable the site and restart of Apache:

$  a2ensite flaskTest
$  /etc/init.d/apache2 reload

and my application was running!

My new font is Gandhi serif.

I'll let it sit for a while and see how I like it in a few days.

I'd always been interested in Python, but I find it very difficult to learn something by following through a tutorial. I need to have a project to get stuck in to and the few ideas I have had were always a little over-ambitious.

Fresh from my win with my search system (and freshly deflated by The Essay), I decided to give Flask a go.

BEST. THING. EVER!

I am so impressed with how quickly I got to grips with Flask, Jinja2 and of course Python. In the last 4 days I've almost completely replicated my staticDimension blog publishing system in Python!

It's still a long way from being shareable in full (I'm not even using it to write this yet, it's still safely locked away in my testing environment) but I wanted to share a small bit of the problem I was up until midnight last night solving.

At the point I wrote this bit, I was feeling really pleased with myself that I'd managed to transition into a whole new language and was already writing simple, efficient, beautiful code. I needed to implement the code for generating the archive pages that make the post date directories browsable.

This is what I came up with:

def rebuild_archive_indexes():
    archiveYearDirs = glob("%s/[0-9][0-9][0-9][0-9]" % app.config['SITE_ROOT_DIR'])
    archiveFile = codecs.open("%s/archive.html" % (app.config['SITE_ROOT_DIR']),
                                               encoding='utf-8', mode='w')

    archiveYears = []                          
    for y in archiveYearDirs:
        year = os.path.split(y)[1]
        archiveYears.append((year, year))

        for (path, dirs, files) in os.walk("%s/%s" % (app.config['SITE_ROOT_DIR'], year)):
            archiveIndex = codecs.open("%s/index.html" % path, encoding='utf-8', mode='w')

            sPath = path[len(app.config['SITE_ROOT_DIR'])+1:]
            if not dirs:
                files.remove('index.html')
                items = map(get_full_title, files)
            else:
                if '/' in sPath:
                    days = dirs
                    month = [sPath.split('/')[1] for x in range(len(days))]
                    yr = [year for x in range(len(days))]
                    items = map(day_name, days, month, yr)
                else:
                    items = map(month_name, dirs)

            archiveIndex.write(render_template("%s/archive.html" % app.config['SITE_TEMPLATE'],
                                          site_root_url=app.config['SITE_ROOT_URL'], 
                                          items=items,
                                          breadcrumb=sPath+'/'))
            archiveIndex.close()

    archiveFile.write(render_template("%s/archive.html" % app.config['SITE_TEMPLATE'],
                               site_root_url=app.config['SITE_ROOT_URL'], 
                               items=archiveYears,
                               breadcrumb=""))
    archiveFile.close()
    flash('Archive rebuilt')
    return redirect(url_for('control_panel'))

def month_name(monthNumber):
    months = ("January", "February", "March", 
                     "April", "May", "June", "July", 
                     "August", "September", "October", 
                     "November", "December")
    return (monthNumber, months[int(monthNumber)-1])

def day_name(dayNumber, month, year):
    d = datetime.date(int(year), int(month), int(dayNumber))
    return (dayNumber, "%s, %s" % (d.strftime('%A'), dayNumber))

def get_full_title(f):
    return (f, f)

I'm pretty sure this would be classified as an abomination and get me kicked out of any Python clubs - month = [sPath.split('/')[1] for x in range(len(days))] to generate a list of the same values just so I can use map() on the days tuple to work out what day the 1st of May was in 2011. get_full_title() isn't fully implemented yet and may well spawn a post of its own too - the lazy method means opening and reading every plaintext blog post (again) just to get the first line. I'm pretty sure I can do better than that.

There's a section of code from my post yesterday that left me feeling uncomfortable:

    $results = array_count_values($results);
    asort($results);
    $results = array_reverse($results);

The middle line was $results = asort($results); the first time around. I always go there first, until I remember that asort() does the sort 'in place'. It's always grated on me.

I was pleased that "eevee" closed with this:

If you only know PHP and you’re curious to learn something else, give the Python tutorial a whirl and try Flask for the web stuff.

I quite like the look of Flask, I might give it a go.