Playing with data

.cvs file

SetDirectory["/Users/mahoney/Desktop"];

data = Import["survey.csv"];

data // TableForm

?Select

get_heights[data_] := 
        Module[{},
  
];

data[[All, 2]]

Dimensions[data]

data[[Range[2, 87], 2]]

average age ?

men's average age ?

women's average age?

how many women are under 5 feet tall ?  Over 6 feet tall ?

Count how many women fall in "bins" of possible heights.  Draw a bar plot (histogram) that shows how many of each height.

Counting words in a text file

text = Import["Ishmael.txt"];

text

FullForm[text]

?*String*

Can you write a function that extracts individual words?

Is there another function that will make this easier - reading in the whole file will give a big text string.
See "Import".

?*Import*

words = Import["Ishmael.txt", Words];

What does that error mean?

words  =  Import["Ishmael.txt", "Words"];

FullForm[words]

How do we remove the puncuation characters ?

Counting the words - how do we find the unique ones?  

Mathematica has a "Union" operation that can do this.  

Each unique word will have an index - where it is in the array.  Can you write a function to find that index?

We'd like to count how many times each word occurs.  So - let's have another array be the counts.  All we need to do is 
increment the correct index.    What should we do about the error if the count array isn't big enough?

Another way : store { {word,count}, {word,count}, ... }

Is this easier with numbers?

r  =  Table[ Random[Integer, {1, 10}], {30}]

count = Table[0, {10}]

What does this next line do ?

Do[ count[[ r[[i]] ]]++ ,  {i, 1, 
        Length[r]}]

count

<< Graphics`Graphics`

BarChart[count]

The reason that it's easier with the numbers is that each number is it's own index - that is, given a number n we can jump right to count[[n]] without having to look up where it should go.

We can do the same kind of thing with the words, if we can find a function that sends each word to a unique number.  Such a function is called a "hash function".

hashFunction[word_]:=Module[{},
];

\(\(\( (*\ 
      A\ try\ at\ a\ hash\ function\ for\ the\ words\ *) \)\(
\)\(hashFunction[word_] := 
        Module[{}, 
value = 
            0; 
Do[
char = \ 
              StringTake[
                word, {i, 
                  i}]; 
Print[\ " char is "\  <> \ 
                ToString[char]];
, {i, 1, 
              StringLength[word]}];
];\)\)\)

hashFunction["Puppy"]

(* 
      A try at a hash function for the words *) 
hashFunction[word_] := 
        Module[{}, 
value = 
            0; 
Do[
char =  
              StringTake[
                word, {i, 
                  i}]; 
value  =  value + 
, {i, 1, 
              StringLength[word]}];
];

?*Char*

ToCharacterCode["a"]

ToCharacterCode["hello"]

Total[{1, 2, 3}]

OK, so all we need to do to get them as characters is to add 'em all up somehow, 
then shrink it down into some range big enough that collisions are unlikely.

1st try : Total[ToCharacterCode[word]]

or do something tricker, then take mod n for some prime n as big as your table.