Tuesday, May 13, 2014

Elixir - recursive max

defmodule MyMax do
  
  defp _max([head], mem) when mem < head, do: head
  
  defp _max([_head], mem) , do: mem

  defp _max([ headl | [ headr | tail ]], mem) when headl < headr and mem < headr, do: _max([ headr | tail ], headr)
  
  defp _max([ _headl | [ headr | tail ]], mem), do: _max([ headr | tail ], mem)
  
  def max([ head | tail ]), do: _max([ head |tail], head)


end


IO.inspect MyMax.max([1, 2, 3, 4, 6])



Beautiful, atl least to me. This is an exercise from the great Programming Elixir book I'm currently reading

Tuesday, April 22, 2014

The Horror

Since I can't say all the things I think to annoying people to their faces, especially those annoying-undeclared-but-effectively-myboss kind of people, here is a compilation of some of the stuff my ears had to bear in my current workplace:


  1. "This is a good idea ! but why didn't you go for it before"
  2. "This is a good idea ! I talked about it with X few months ago"
  3. "I played around with some library you referred to me in the past" (but when I suggested to use it, it was brutally rejected)
  4. "Don't you know this/that/very-known-tech-blogger ? very bad, you should follow him"
  5. "We don't ask technical questions in our interviews. Read the very-known-tech-blogger's blog. He is against technical questions in technical interviews" (Of course, we can't think for ourselves)
  6. Me: so you are saying we don't need to see a single line of code of the person we are interviewing ? It: is she supposed to know to write code ? I should read her CV (about a Data Scientist Position candidate)

Wednesday, April 9, 2014

Apache Pig - Using Compression


Subtitle: this post is dedicated to all the warriors trying to fight Pig to do what they want (that is, anything beyond the trivial JFFGLS (join-foreach-filter-group-load-store) without having proper documentation.

For the good heavens in the sky. This is how to set compression in Apache Pig 0.11.1.1 as compiled by Amazon (the one supplied on EMR):

To enable compression of results (when writing your final part file results), place these before you call STORE:

       

set output.compression.enabled     true
set output.compression.codec   com.hadoop.compression.lzo.LzopCodec

 

This should produce part-*-*.lzo files

Reading them in another Pig job does not need anything but the regular LOAD USING statements since Pig detects (I'm guessing by the .lzo suffix) that these are compressed files.

Of course, since all the above is related to EMR, the codecs are there (as well as Snappy), preinstalled.

Oddly, and this is yet an unsolved mystery for me, using compression on the writer, with PigStorage('\t', '-schema') doesn't work when the data is being read by another job: It produces "java.lang.IndexOutOfBoundsException: Index: 19, Size: 19" exceptions which shoots the job into the heavens. This happened with all the compressions types I've tryed including bz2, lzo and snappy. Took me a while to figure out that it's not the compression to blame, but the '-schema' directive which doesn't apparently go well with compression.

And hey don't worry, all of the above is not documented anywhere. I invite my followers (hi - currently nobody) to comment and correct me if I'm wrong. And another thing, hi Google - you keep track of everything I do, and you know by now I'm jewish. When I google for "pig", please don't refer me to pig farms, growers, etc. Another thing Google - Pig is now at version 0.12.0, why do you keep giving me the pig docs of version 0.7.0 ?