Clustering Big Data

Join Us for a discussion on Clustering Big Data

When: Thursday June 4, noon – 1:00 pm EST. 
Where: (Virtual Meeting) Contact Us for coordinates.

Description: Approximate Nearest Neighbor methods for clustering and indexing have been actively researched ever since the K-Means algorithm was published in 1975 (and coded in FORTRAN). A recent book lists about 300 variants and related topics.

The 50th Anniversary issue of Communications of the ACM in 2008 cited two pieces of "Breakthrough Research". One was MapReduce, the other was clustering based on Locality Sensitive Hashing (LSH). Locality Sensitive Hashing is for sets of large data and alleviates many of the issues seen with k-means. Want to see if a body of code has remarkable similarity to a public github repo? Want to see "similar" fragments of DNA that are common between several species? LSH will get you there faster than most other techniques.

The talk will demonstrate OpenLSH, an open source implementation of LSH we have been working on.

Speaker Bios: Dr. J Singh is a Principal at, a Cloud and Big Data consulting company. He is a frequent speaker on NoSQL, Hadoop--Map/Reduce and analytics of social media. He is the originator ...

OpenLSH - a framework for locality sensitive hashing

Presentation at Pivotal IO Meetup in New York

Mining Social Media for Marketing

There is a wealth of information tied up in social media and it is up to your Marketing organization to unlock its potential.

We recently ran a workshop on Facebook Analytics using Elastic Map/Reduce. In case you were unable to attend, slides from that presentation:
Facebook Analytics with Elastic Map/Reduce from J Singh

Big Data Laboratory

So you want to get going on having your organization come up to speed on Big Data. What next?

Your first step could be to set up a sandbox for the participants. We set up ours on Amazon EC2 because Amazon hosts a number of public data sets, as do InfoChimps and others. These data sets are convenient for initial experimentation.

It turns out that getting access to the data where you can touch it and feel it is the easy part. Christopher Miles takes an inspiring stab at the problem in his blog post All Your HBase Are Belong to Clojure

Our approach is a bit different, using Google App Engine's Map Reduce to let us focus on the task at hand. This approach doesn't work for all situations but can shave valuable time off the learning curve in some cases.

I had occasion to develop this idea further at a recent talk to the Google App Engine group. Slides from that talk:
Big Data Laboratory from J Singh

The Hadoop Ecosystem

Here's a more detailed outline of my talk on March 12. To make the talk more relevant to you, if you have a use case you'd like me to discuss, we'd love to hear about it, and possibly incorporate it into the talk. Join us for ... (see the end of this post).

If you came here looking for the presentation, here it is.


  1. What Hadoop is, and what it's not
  2. Origins and History
  3. Hello Hadoop, how to get started.

The Hadoop Bestiary

  1. Core: Hadoop Map Reduce and Hadoop Distributed File System (HDFS)
  2. Data Access: HBase, Pig and Hive
  3. Algorithms: Mahout
  4. Data Import: Flume, Sqoop and Nutch

The Hadoop Providers

  1. Apache
  2. Cloudera
  3. What to do if your data is in a database

The Hadoop Alternatives

  1. Amazon EMR
  2. Google App Engine
For those that weren' t able to attend, here is the presentation:

Hands on with Hadoop

This post begins a series of exercises on Hadoop and its ecosystem. Here is a rough outline of the series:
  1. Hello Hadoop World, including loading data into and copying results out of HDFS.
  2. Hadoop Programming environments with Python, Pig and Hive. 
  3. Importing data into HDFS. 
  4. Importing data into HBase. 
  5. Other topics TBD.  
We assume that the reader is familiar with the concepts of Map/Reduce. If not, feel lucky you have Google. Here are two of my favorite introductions: (1) The Story of Sam and (2) Map Reduce: a really simple introduction.

Hello Hadoop World

We build this first environment on Amazon EC2. We begin with ami-e450a28d, an EBS-based 64-bit ubuntu machine that's pretty bare-bones. To run Hadoop, we need at least an m1.large instance — using a micro instance is just too painful.

For the actual exercise, we will use the excellent tutorial by Michael Noll. Michael first wrote it in 2007 and has kept it up to date. The tutorial helps you install map/reduce and use it for computing the count of every word in a large text. Michael's post is pretty much right on. Things to keep in mind:
  • Make sure python 2.6 ...

Stop the Data Warehouse Creep

Have you experienced this?
  • Your data warehouse takes about 8 hours to load the cube. The cube is updated weekly; 8 hours per week is a small price to pay and everyone is happy.
  • It is missing a few data elements that another group of analysts really needs. You add those and now the cube takes 9 hours to load. No big deal.
  • Time passes, the business is doing well, the size of the data quadruples. It now takes 36 hours per week to load the cube.
  • Time passes, some of the data elements are not needed any more but it is too hard to take them out of the process — it continues to take 36 hours.
  • You add yet another group of analysts as user, a few more data elements, it now takes 44 hours per week!
  • You get the picture… the situation gets more and more precarious over time.

We hope the dam holds! 

Here's an idea from Read Write Web: 

One example … was a credit card company working to implement fraud detection functionality. A traditional SQL data warehouse is more than likely already in place, and it may work well enough but without enough granularity for ...

Sources of Noise in Big Data

Curt Monash published a trio of articles about dirty data last summer where he cited use cases for storing dirty data even when the signal-to-noise ratio is pretty low. He had this observation about the value of the data in even in cases where the signal-to-noise ratio is low:

Intelligence work is one case where the occasional black swan might justify gilded cages for the whole aviary; the same might go for other forms of especially paranoid security.

Big Data can be dirty and the collection process must pay attention to data quality. This is different from the type of data we are used to in your typical database, with all the focus on data integrity. The point is that even data with very low integrity has valuable nuggets of information. 

Whatever happened to data integrity? The RDBMS people mastered the art of data integrity long ago and along come these NoSQL folks who just ignore all that learning. This data is junk!

Human input is often a source of noise in the data. Big data is used for analyzing Twitter feeds, blog posts, comment forums, chat sessions and the like and text data can be ambiguous. It's context ...

Executive Briefing Promotion

We will be giving away one free ticket to the executive briefing. To be entered into the drawing for it, please let us know why you would like to attend and what you expect to get out of the briefing.

The winner will be drawn at random from the attendees and will be announced at the briefing. To be considered for the free ticket, please submit your response by February 26, 2012.

Click here to let us know.

Big Data Executive Briefing

DataThinks' Executive Briefing on Big Data, NoSQL and Data Analytics is scheduled for March 1, 2012. Signup Information below.

These are are some of the topics we intend to cover in that briefing.
  • What is Big Data and what problems does it solve? What opportunities does it present that weren't available before?
  • These terms are often seen in articles about Big Data: NoSQL, Map Reduce, Analytics, Schema-less Databases. How do they relate to each other and how do they differ?
  • What problems you are better off solving with traditional database solutions?
  • The various types of NoSQL databases and what are they each appropriate for? 
    • Key-value stores (e.g., Riak, Redis, Voldemort, Tokyo Cabinet)
    • Document stores (e.g., MongoDB, CouchDB)
    • Wide column stores (e.g., Cassandra, Hypertable)
    • Graph Databases (e.g., Neo4J)
  • Analysis and Visualization
    • What is Map / Reduce and the opportunities it presents for your business.
    • The cost / performance trade-offs  of running Map Reduce in the cloud, on Amazon EC2 or on Google, or in-house.
Any additional topics you would like to see covered? Please leave a comment.

The event is closed but corporate briefings based on this material are available. Please contact us to arrange.