Thursday, October 08, 2015

How the mind works

I have been reading this fascinating book by Steven Pinker for some time. It is large – 765 pages – and both engrossing and intellectually deep. Thus my normal high reading speed is tempered somewhat. In a sense, this book provides a great deal of measured academic background for David Lodge's excellent novel, "Thinks". There is a section on qualia and one section on emotions and why they exist. Whilst the statement apparently made by Darwin and discussed at length by Lodge's characters – crying is a puzzler – does not appear in Pinker's book (or if it does, I haven't come across it yet), there is a section about grief, which also is discussed by Lodge's fictional professor, Ralph Messenger. 

Following is some of what Pinker writes on the subject: No one knows what, if anything, grief is for. Obviously the loss of a loved one is unpleasant, but why should it be devastating? Why the debilitating pain that stops people from eating, sleeping, resisting diseases, and getting on with life? Jane Goodall describes a young chimp, Flint, who after the death of his beloved mother became depressed and died himself as if of a broken heart. [I should mention here that after my paternal grandmother died, my paternal grandfather died after a few weeks, and the same thing recently happened to the parents of someone on the kibbutz – NBN].

Some have suggested that grief is an enforced interlude for reassessment. Life will never be the same so one must take time to plan how to cope with a world that has been turned upside down. Perhaps grief also gives people time to contemplate how a lapse of theirs may have allowed the death and how they might be more careful in the future [pp 444-5].

Another snippet reminds me of a letter which I received from my mentor a few weeks ago which incensed me at the time; I decided to wait a few days to calm down before replying. When I did reply, his letter seemed much more to the point and basically exposed my misunderstanding. Pinker writes: Sometimes we have glimpses of our own self-deception. When does a negative remark sting, cut deep, hit a nerve? When some part of us knows it is true. If every part knew it was true, the remark would not sting; it would be old news. If no part thought it was true, the remark would roll off; we could dismiss it as false. Travers recounts an experience which is all too familiar (at least to me). One of his papers drew a published critique, which struck him at the time as vicious and unprincipled, full of innuendo and slander. Rereading the article years later, he was surprised to find that the wording was gentler, the doubts more reasonable, the attitude less biased than he remembered. Many others have made such discoveries; they are almost the definition of "wisdom". [p 447]

Saturday, October 03, 2015

More statistics functions with SQL

There are  three  commonly  used  measures  of  central  tendency: the  mode, the median and the mean. The mode describes the most common score, the median the score of the middle case, and the mean the average score. The measures available for analysis depend on the type of variable.


 In my data, the 'results' table contains the results (naturally) for a given section (or variable). Since data from nominal and ordinal variables cannot be aggregated (see previous post), each section corresponds to a single question, thus the results (or scores) are actually the options chosen for that question.

In my pilot study, the results for the following ordinal question were
Q45: What is your highest level of education?
1. Ninth grade (O-levels) - 0
2. A levels - 1
3. Professional diploma - 3
4. Bachelor degree - 8
5. Masters degree - 4
6. Doctorate - 0

The mode is clearly 8 - bachelor degree. This can be determined by a fairly easy SQL query:
select score, count (*) from results where section = :p1 group by score order by count (*) desc
where p1 is a parameter whose value is the number of the section being evaluated (for education, it's 12). The query will return several rows, depending on how many options exist for the question, but only the first row (which will be 4, 8) would be used.

If one arranges the values by score, they are 1333444444445555. As there are 16 values, the median lies between the 8th and 9th value, which in both cases is 4. Obtaining the median via SQL without using a cursor is fairly tricky - it involves the use of two less frequently used SQL keywords: first and skip. The query which I came up with is
select first 1 skip 8 score from results where section = :p1 order by score
The number following first defines the number of rows to be returned - in this case, one. The number following skip defines how many rows in the original dataset should be skipped before applying first: translated into English, the query says order the values, skip the first eight values then return the next one. In other words, return the ninth value - the median - when the rows are ordered.

The reason that the '8' is on a separate row is because it cannot be parameterised; it has to be added dynamically into the SQL query during preparation; isolating this value simplifies the addition.