Sharding (refresher)

Scaling is an eventuality with any growing business. It comes in many forms. For the online business, database scaling is the first big technical hurdle.

The new generation of webapps demand a near-persistent connection with the database, doing multiple read/writes over a single session; the more LIVE data you throw, the longer "they" stick with you. Couple that with the "viral" aspect, and you know that soon as your app gets noticed, the database demands are gonna surge. Databases are the foremost when it comes to contingency planning.

DBs are moving to the cloud, much like our individual data. Clouds, for all the good they offer, aren't strong when it comes to sustained I/O ops.
A 0.9GB database takes 1 min to load.
A 30GB database takes 10 days to load...

Another aspect is that the newer-gen NoSQL DBs are single threaded. Which means the CPU is only handling a single op at a time. In case of a lock, it could take a long time to resolve. Everyone (communicating with the DB) takes the brunt.

Horizontal partitioning is a design principle whereby rows of a database table are held separately, rather than splitting by columns (as for normalization). Each partition forms part of a shard, which may in turn be located on a separate database server or physical location. The advantage is the number of rows in each table is reduced (this reduces index size, thus improves search performance). If the sharding is based on some real-world aspect of the data (e.g. European customers vs. American customers) then it may be possible to infer the appropriate shard membership easily and automatically, and query only the relevant shard.

Another place sharding can be used is to reduce contention on data entities. It is especially important when building scalable systems to watch out for those piece of data that are written often because they are always the bottleneck. A good solution is to shard off that specific entity and write to multile copies, then read the total. An example of this "sharded counter wrt GAE: http://code.google.com/appengine/articles/sharding_counters.html

If you have queries to a DBMS for which the locality is quite restricted (say, a user only fires selects with a 'where username = $my_username') it makes sense to put all the usernames starting with A-N on one server and all from M-Z on the other. By this you get near linear scaling for some queries.

Resources
[1] http://en.wikipedia.org/wiki/Shard_(database_architecture)
[2] http://stackoverflow.com/questions/992988/what-is-sharding-and-why-is-it-important
[3] http://www.slideshare.net/rightscale/rightscale-webinar-scaling-your-database-in-the-cloud
[4] https://vimeo.com/32541189

Fight Spam, but how?

Been inundated with spammy referrers in the last few months. I'm sure I'm not the only one. I know what it is - backlink spam [1, 2], which means those dirty ad networks generating fake clicks to blogs, to be displayed in the backlink/trackback list, which in turn adds to their PageRank on Google. So wily are these spammers that instead of using their direct URL they are now masking it through t.co and then further wrapping them into StumbleUpon or Facebook or [insert any respectable sounding name] redirects. The blog owners will both be confused and curious - "Gee, who could be linking to me from http://www.filmhill.com/redirect.php?url=http%3A%2F%2Ft.co%2FUEmjqui3"

Sadly, even though I have turned my backlinks off through Blogger settings, these guys won't stop. The TLDs are too diverse to block out. But one common pattern observed among these links are that all use t.co to masquerade. Probably could insert a JS snippet to block any incomings of this nature. But I don't see how that solves anything - Google Analytics will still register the referral since the page is going to be accessed and its going to show up anyways.

97 things

I'm reading a book called "97 things a programmer should know". After the first few pages, it seemed like common sense, so I put it down. It didn't solve my crisis of the stalemate I find myself in. Not only that, it was generally uninteresting.

What I do want to read, however, would be "97 things a programmer wouldn't know", which could introduce some new paradigms to the art of software programming that an average programmer won't be familiar to - after years of being in the industry, I have seen people decaying and falling out-of-date; or "97 things a programmer should forget", which could be a treatise against the prevalent bad practices that every new guy falls into.

Next I pit hopes on "Beautiful Architecture", which I understand as an effort towards creating a "programmauteur", my ideal.

State-of-the-art Vertical Scroll Sites

http://benthebodyguard.com/ http://2011.beercamp.com/ http://sf-langstrasse.hvo.ch/page/ http://activatedrinks.com/

Official Google Blog: When patents attack Android

Official Google Blog: When patents attack Android: "I have worked in the tech sector for over two decades. Microsoft and Apple have always been at each other’s throats, so when they get into ..."

Gender Connection

THE MALE PLUG

THE FEMALE RECEPTACLE

Learn More

Notifications in Chrome

Some of us would be familiar, and piqued, by the recent suggestion in our Gmail windows: "Click here to enable mail notifications in Chrome". Choose to, and whichever tab you might be browsing on, any new mail or chat message is reported in a tiny corner bottom right of the screen. This nifty addition surprised me enough to find out more about the concept, and its other possible applications.

Notifications to the desktop are nothing new, but notifications on the browser are.
The OSX folks have had something called Growl notifications in place, which requires installation, then could be accessed through the windows.fluid object built into Safari on Mac.

Google, with their support for Web Notifications, and their W3C recommendation in general, have tried standardizing this concept.
Here's a small working example for the same:

PS: Try it, in baby steps, on Jash. FUN!

<script type="text/javascript">
(function() {
var notificationStatus;

setupNotifications = function() {
if(window.webkitNotifications) {
if(window.webkitNotifications.checkPermission() == 0 ) { //permission granted
notificationStatus = 'Permission granted';
} else if(window.webkitNotifications.checkPermission() == 1 ) { //permission unknown
//notificationStatus = 'Permission not granted yet';
notificationStatus = "Unknown, click to <a href='javascript:window.webkitNotifications.requestPermission(setupNotifications);'>enable web notifications</a>.";
window.webkitNotifications.requestPermission(setupNotifications);
} else { //permission denied
notificationStatus = 'Permission denied';
}
} else alert('Not supported');

if(!!document.getElementById("notificationStatus"))
document.getElementById("notificationStatus").innerHTML = notificationStatus;
}

notifyMe = function() {
//permission already granted
if(window.webkitNotifications.checkPermission() == 0 ) {
var popup = window.webkitNotifications.createNotification(
"myicon.ico",
"You nailed it",
"My good sir, by delving into this notification thingy you open yourself to plenty of bobfoolery"
);
//ondisplay is one of the three events for the notification API
// ondisplay, onerror, oncancel
popup.ondisplay = function() {
setTimeout(function () {popup.cancel();}, 6000);
//we keep a timeout for the notification, since they don't have one by default
};
popup.show(); //this displays the notification
}
}
window.onload = setupNotifications;
})();
</script>

A simplified concept, a small API, and user security considerations - the spec has its bases covered. Presently these notifications only work on the Chrome browser across Windows and Linux.
For the old crowd, things were ridiculous, with the programmers having to resort to annoying and invasive hacks like window.alert. This new feature would give online tools like e-mail clients, calendering software, task managers, monitoring systems, etc. an elegant and consistent way to notify users.

However, not everybody is gaga over it. After some toying around, even I am a bit skeptical about the specs and the architecture: The notifications take a while to associate with their tab. They aren't a welcome experience to deal with when your friends start firing messages in your gmail chat window. Moreover, with the proliferation of these (among Tier 1 sites, to merely begin with) we can only anticipate an avalanche of these, since the recommendation suggests no way to restrict or control the notifications.