Skip to main content

GAE: Batch operations

Given the limit on individual requests with GAE, doing things like batch updates to your database (live) can be an interesting challenge. I recently had to update all of the votes for Challenge-You! to support a new rating system. The best approach I found was an AJAX-style solution. Have one request that returns a list of references to the entities you need to change (or if you're working with more than 1000 entities, you might want to partition the list into chunks by date or some other order, and work with one chunk at a time). Keeping those references in a stack in Javascript, pop them off one-by-one and send each to the controller that applies the change, repeating until the stack is empty. If the operations aren't too expensive, you could even send a handful of them per-request to speed things up. It's also trivial to implement some kind of progress bar or other graphical update to keep you informed.

And example of the Javascript code (using jQuery) that handles the client side:
var data = null;

function doupdate() {
if(data.length == 0) return;
$.ajax({
type:"POST",
url:"doupdate",
data:"id=" + data.pop(),
success:function(msg) {
setTimeout(function() {
doupdate();
}, 1000);
}
});
}

$(document).ready(function() {
$.ajax({
type:"POST",
url:"getlist",
data:"",
success:function(msg) {
data = msg.split(" ");
doupdate();
}
});
});

In this example, my site would have two urls for handling this, "getlist" which returns a space-delimited list of references/ids (however you're handling the entities) and "doupdate" which takes in one reference and applies the needed change.

Comments

Ali said…
I was also met with this ghastly limitation of GAE, and the "Warning you are using over your allotted amount of air for breathing" was scaring me.

I also solved it in a similar way, by splitting the requirements into chunks and calling it over ajax. But what a pain!

Popular posts from this blog

Procedural music with PyAudio and NumPy

Combining two of my favorite pastimes, programming and music... This is the hacky "reduced to it's basic components" version of a library I've been working on for generating music and dealing with music theory.

Tweaking the harmonics by changing the shape of the harmonic components and ratios can produce some interesting sounds. This one only uses sine waveforms, but a square / saw generator is trivial with numpy.

It takes a second to generate, so don't turn your volume up too loud in anticipation (it may be loud).

import math
import numpy
import pyaudio
import itertools
from scipy import interpolate
from operator import itemgetter


class Note:

NOTES = ['c','c#','d','d#','e','f','f#','g','g#','a','a#','b']

def __init__(self, note, octave=4):
self.octave = octave
if isinstance(note, int):
self.index = note
self.note = Note.NOTES[note]
elif isinstance(note, st…

Build a Feed Reader in Python (Parts 7-9)

Part 07 Adding Jinja2 templates to a flask web application.

 Part 08 Adding static files so we can serve some CSS to style our app.

Part 09 Adding a background task to continuously update the articles while the application is running.

Write a Feed Reader in Python

I just started a new video tutorial series. This time it'll cover the entire process of writing an RSS feed reader in Python from start to finish using the feedparser module, flask, and SQLAlchemy. Expect to see about 3-4 new videos a week until this thing is finished!
Click to watch