Subtitle: this post is dedicated to all the warriors trying to fight Pig to do what they want (that is, anything beyond the trivial JFFGLS (join-foreach-filter-group-load-store) without having proper documentation.
For the good heavens in the sky. This is how to set compression in Apache Pig 0.11.1.1 as compiled by Amazon (the one supplied on EMR):
To enable compression of results (when writing your final part file results), place these before you call STORE:
set output.compression.enabled true
set output.compression.codec com.hadoop.compression.lzo.LzopCodec
This should produce part-*-*.lzo files
Reading them in another Pig job does not need anything but the regular LOAD USING statements since Pig detects (I'm guessing by the .lzo suffix) that these are compressed files.
Of course, since all the above is related to EMR, the codecs are there (as well as Snappy), preinstalled.
Oddly, and this is yet an unsolved mystery for me, using compression on the writer, with PigStorage('\t', '-schema') doesn't work when the data is being read by another job: It produces "java.lang.IndexOutOfBoundsException: Index: 19, Size: 19" exceptions which shoots the job into the heavens. This happened with all the compressions types I've tryed including bz2, lzo and snappy. Took me a while to figure out that it's not the compression to blame, but the '-schema' directive which doesn't apparently go well with compression.
And hey don't worry, all of the above is not documented anywhere. I invite my followers (hi - currently nobody) to comment and correct me if I'm wrong. And another thing, hi Google - you keep track of everything I do, and you know by now I'm jewish. When I google for "pig", please don't refer me to pig farms, growers, etc. Another thing Google - Pig is now at version 0.12.0, why do you keep giving me the pig docs of version 0.7.0 ?
No comments:
Post a Comment