Skip to main content

GAE: Batch operations

Given the limit on individual requests with GAE, doing things like batch updates to your database (live) can be an interesting challenge. I recently had to update all of the votes for Challenge-You! to support a new rating system. The best approach I found was an AJAX-style solution. Have one request that returns a list of references to the entities you need to change (or if you're working with more than 1000 entities, you might want to partition the list into chunks by date or some other order, and work with one chunk at a time). Keeping those references in a stack in Javascript, pop them off one-by-one and send each to the controller that applies the change, repeating until the stack is empty. If the operations aren't too expensive, you could even send a handful of them per-request to speed things up. It's also trivial to implement some kind of progress bar or other graphical update to keep you informed.

And example of the Javascript code (using jQuery) that handles the client side:
var data = null;

function doupdate() {
if(data.length == 0) return;
$.ajax({
type:"POST",
url:"doupdate",
data:"id=" + data.pop(),
success:function(msg) {
setTimeout(function() {
doupdate();
}, 1000);
}
});
}

$(document).ready(function() {
$.ajax({
type:"POST",
url:"getlist",
data:"",
success:function(msg) {
data = msg.split(" ");
doupdate();
}
});
});

In this example, my site would have two urls for handling this, "getlist" which returns a space-delimited list of references/ids (however you're handling the entities) and "doupdate" which takes in one reference and applies the needed change.

Popular posts from this blog

Exploring Sub-GHz Radio With RTL-SDR and GQRX

Today I took a look at some hardware for working with sub-GHz radio transmissions, especially FM. All of which came in a previous HackerBoxes box. My favorite item was definitely the RTL-SDR dongle! I could play around with this thing for hours surfing the radio waves and listening to weird broadcasts. There's even a Python library named pyrtlsdr for working with these devices so you can scan/record radio transmissions programmatically:  https://github.com/roger-/pyrtlsdr Next I'm going to solder together the FM transceivers included in the box and play around with writing Arduino firmware for them. Those things should work similar to the LoRa modules I've been using but they trade in the long range for increased bandwidth. Should be good for streaming data locally and still have much better range than most home WiFi or Bluetooth. Here's a video overview of the hardware included in HackerBox 0034 as well as a demo of the RTL-SDR dongle using GQRX to visually and...

Always Secure Your localhost Servers

Recently I was surprised to learn that web browsers allow any site you visit to make requests to resources on localhost (and that they will happily allow unreported mixed-content ). If you'd like to test this out, run an HTTP server on port 8080 (for instance with python -m http.server 8080 ) and then visit this page. You should see "Found: HTTP (8080)" listed and that's because the Javascript on that page made an HTTP GET request to your local server to determine that it was running. Chances are it detected other services as well (for instance if you run Tor or Keybase locally). There are two implications from this that follow: Website owners could potentially use this to collect information about what popular services are running on your local network. Malicious actors could use this to exploit vulnerabilities in those services. Requests made this way are limited in certain ways since they're considered opaque , meaning that the web page isn't able...

DIY Solar Powered LoRa Repeater (with Arduino)

In today's video I be built a solar powered LoRa signal repeater to extend the range of my LoRa network. This can easily be used as the basis for a LoRa mesh network with a bit of extra code and additional repeaters. Even if you're not into LoRa networks all of the solar power hardware in this video can be used for any off-the-grid electronics projects or IoT nodes!