Sunday, January 1, 2012

Hbase - Thrift - Python: The missing bits

A short how-to-install Thrift.0.8.0 on Fedora 16, for usage with Thrift-python and Thrift-java - that is, you would be able to use Thrift from Python and Java. The Hbase/Thrift stuff is based on http://wiki.apache.org/hadoop/Hbase/ThriftApi

1) Install the following dependencies:
sudo yum -y install boost.x86_64 flex ruby make automake gcc gcc-c++ kernel-devel byacc
2) Download thrift source tar.gz:
wget http://www.apache.org/dyn/closer.cgi?path=/thrift/0.8.0/thrift-0.8.0.tar.gz .
3) Unpack and cd to the created directory

4) Build Thrift from its sources:
 ./conifgure && make && sudo make install

5) The previous step installs the needed python packages in the correct Python site-packages, so that eventually you can  :
from thrift import thrift
6) To actually use thirft, which acts as a bridge between programming languages, you've got to have some sort of mapping, or put it better, some sort of markup which is language neutral and defines the types Thrift would use to serialize and deserialize you messages to the Thrift server. In Hbase's case, this mapping is the specified in the Hbase.thrift file located at ${thrift_home}/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift .

7) Now, to actually 'talk' to the Thrift server in your favorite language, in this case Python, you need to generate Python sources that would map to the Hbase.thrift types. This is done by the following call:

thrift --gen py ${thrift_home}/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift
8) Now you would have a directory with Python sources named 'gen-py'.

9) Just add the generated folders to the PYTHONPATH (or sys.path.append('gen-py')) yo use the generated classes.

... This concludes the installation fun.


And now for a nice script I have made to import data into Hbase using Thrift-Hbase-Python trio:
from thrift.transport.TSocket import TSocket
from thrift.transport.TTransport import TBufferedTransport
from thrift.protocol import TBinaryProtocol
from hbase import Hbase
from hbase import ttypes
from hbase.Hbase import Client, ColumnDescriptor, Mutation, BatchMutation
import time 
import datetime

transport = TBufferedTransport(TSocket("localhost", 9090))
transport.open()
protocol = TBinaryProtocol.TBinaryProtocol(transport)

client = Hbase.Client(protocol)
t = "sourcetable"
columns = []
col = ColumnDescriptor()
col.name = 'core:'
col.maxVersions = 10
columns.append(col)

columns.append(col)

try:
  print "creating table: %s" %(t)
  client.createTable(t, columns)
except ttypes.AlreadyExists, ae:
  print "WARN: " + ae.message

f = open("/home/amit/Downloads/matching_4.txt")
headers = f.readline()
headers = headers.split("\t")
headers = map(lambda x: x.strip(), headers)
bom = "\xef\xbb\xbf"
headers = map(lambda x: x.replace(bom, ''), headers)

batchMutations = []

for i, line in enumerate(f):
  lines = line.split("\t")
  lines = map(lambda x: x.strip(), lines)
  if len(lines) != len(headers):
    print lines
    break
  mapped = dict(map(None, headers, lines))
  
  timestamp = mapped['timestamp_update']
  if (timestamp.strip()):
    timestamp = timestamp.split("-")

    timestamp = (map(int,timestamp))
    tt = datetime.date(timestamp[0], timestamp[1], timestamp[2])
    tfromepoch = time.mktime(tt.timetuple())
    print(tfromepoch)
  mutations = [Mutation(column="core:%s" % k, value=v) for k,v in mapped.items()]
  if tfromepoch:
#    client.mutateRowTs(t, mapped["entity_id"], mutations, long(tfromepoch))
    mutations.append(Mutation(column="core:t", value=str(long(tfromepoch))))
#  else:
#    client.mutateRow(t, mapped["entity_id"], mutations)
  batchMutations.append(BatchMutation(mapped["entity_id"], mutations))
  if (i % 100 == 0):
    client.mutateRows(t, batchMutations)
    batchMutations = []

#flush:
if batchMutations:
  client.mutateRows(t, batchMutations)

transport.close()

First POST

import urllib

params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
f  = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
print(f.read())

This is my way of doing a POST ...
and the experienced reader would quickly note that this is Python. No type deceleration. Free spirit at the price of human irresponsibility.