Skip to main content

GAE: Batch operations

Given the limit on individual requests with GAE, doing things like batch updates to your database (live) can be an interesting challenge. I recently had to update all of the votes for Challenge-You! to support a new rating system. The best approach I found was an AJAX-style solution. Have one request that returns a list of references to the entities you need to change (or if you're working with more than 1000 entities, you might want to partition the list into chunks by date or some other order, and work with one chunk at a time). Keeping those references in a stack in Javascript, pop them off one-by-one and send each to the controller that applies the change, repeating until the stack is empty. If the operations aren't too expensive, you could even send a handful of them per-request to speed things up. It's also trivial to implement some kind of progress bar or other graphical update to keep you informed.

And example of the Javascript code (using jQuery) that handles the client side:
var data = null;

function doupdate() {
if(data.length == 0) return;
$.ajax({
type:"POST",
url:"doupdate",
data:"id=" + data.pop(),
success:function(msg) {
setTimeout(function() {
doupdate();
}, 1000);
}
});
}

$(document).ready(function() {
$.ajax({
type:"POST",
url:"getlist",
data:"",
success:function(msg) {
data = msg.split(" ");
doupdate();
}
});
});

In this example, my site would have two urls for handling this, "getlist" which returns a space-delimited list of references/ids (however you're handling the entities) and "doupdate" which takes in one reference and applies the needed change.

Popular posts from this blog

DIY Solar Powered LoRa Repeater (with Arduino)

In today's video I be built a solar powered LoRa signal repeater to extend the range of my LoRa network. This can easily be used as the basis for a LoRa mesh network with a bit of extra code and additional repeaters. Even if you're not into LoRa networks all of the solar power hardware in this video can be used for any off-the-grid electronics projects or IoT nodes!  

A Lesson in LoRa Module P2P Standards (or the Lack Thereof)

I got a handful of LoRa modules from Reyax a while back, the RYLR896 model based on Semtech SX1276 chips. Instead of using an SPI interface they operate over UART using a small set of AT commands. This made them easier to work with since I didn't have to dig too deeply into a bunch of SPI registers and Semtech specs and they communicate between one another really well. My Espruino JS module for them is available here , which I've used in a few of my YouTube videos. And more recently I've written a MicroPython module for them here .   (A pair of Reyax RYLR896  modules) But, always being on the lookout for different boards and platforms I eventually ended up with a few Maduino LoRa boards. These are cool because they have an Arduino-compatible ATmega328 and the same Semtech LoRa chip (via an RFM95) both integrated on one board. They weren't compatible with Espruino or MicroPython though, and they used the SPI interface instead of AT commands so I knew I would need to lo

Always Secure Your localhost Servers

Recently I was surprised to learn that web browsers allow any site you visit to make requests to resources on localhost (and that they will happily allow unreported mixed-content ). If you'd like to test this out, run an HTTP server on port 8080 (for instance with python -m http.server 8080 ) and then visit this page. You should see "Found: HTTP (8080)" listed and that's because the Javascript on that page made an HTTP GET request to your local server to determine that it was running. Chances are it detected other services as well (for instance if you run Tor or Keybase locally). There are two implications from this that follow: Website owners could potentially use this to collect information about what popular services are running on your local network. Malicious actors could use this to exploit vulnerabilities in those services. Requests made this way are limited in certain ways since they're considered opaque , meaning that the web page isn't able