Thesis crash

.

I will discuss a delicate matter in this blog post and I will therefore pick my words quite carefully and perhaps not tell exactly all there is to tell so as to soften and "anonymize" the issue and protect the specific persons involved. Here is the cast of characters in (order of importance to) the story to be told:

- X: student at our international masters education in media management.

- B: his thesis advisor/supervisor at Stockholm School of Economics (SSE).

- H: his thesis co-advisor/co-supervisor at SSE - the guy who actually did the work of supervising the thesis.

- J: the examiner at KTH/Media technology.

- D: the stand-in examiner at KTH/Media technology who actually read and queried the student at his master's thesis oral presentation/defense. -- This is me.

- T: the thesis administrator at our department who handles practical-administrative questions and routines around the master's theses.

- N: the examiner at KTH/Media technology until the end of 2010.

- BH, A, K, O, CL, CR: Miscellaneous persons who have been involved in trying to give advice and figure out what to do.

Act 1 - Prelude

- J is supposed to be the examiner of a master's thesis, but J is pressed for time as he has gotten tasks from the prefect that urgently needs to be handled. J sends a mail to D and two other persons, asking if someone else can take on the job of being the examiner of two master's theses during the coming week.

- D concurs and accepts the task. The two master's theses unfortunately arrive only on Monday, the day before the oral presentation, and the theses are hefty - around 150 pages to read altogether. D reads one thesis on Monday night and the other on Tuesday morning.

- The oral presentations are ok. The first thesis is ok, but the second thesis seems to have little to do with media technology and the master's degree program in media management in which the student X is enrolled.

- D goes home and ponders the thesis and its contents. The more he thinks about it, the more strange it seems to be presenting a thesis on such an odd (from a media technology perspective) topic. D feels he can not judge the quality of the thesis. Worse, there is no other person that he can hand the thesis over to, because we don't really know anything about the particular subject of the thesis as it has nothing to do with what we do at our department. Both the supervisor and the co-supervisor come from the Stockholm School of Economics (SSE), department of marketing and strategy, and the thesis has a very SSE-feel to it. What to do?

∑ What is going on here???

Act 2 - confusion

- D realizes that there are two questions that need to be answered:

- How could this happen, and what can be done to prevent it from happening again?

- What do we do now - right here and right now with this master's thesis?

- D initiates a facts-finding mission and calls people left and right over the following week, discusses the matter with colleagues at the department and sends mail to others. A degree of partial understanding leads to a number of flawed hypotheses and misunderstandings:

- Supervisor B at SSE stated that "I don't know if we understood that this [was a thesis that] was to be presented at the KTH [Royal Institute of Technology]". Furthermore, even though B is the formal supervisor, it turns out the co-supervisor H has been the de-facto supervisor of the thesis. X has done his thesis in a project that H manages and where several other students have been involved. H is unreachable, away on a trip (relating to the project in question).

- D sends mail to former examiner N who retired during the spring. N would have been the person who saw and approved (?) of X's thesis specification sometime during the autumn. N can unfortunately not find any specification in his computer. Could it have been handed in on paper and got lost in the hustle and bustle of N cleaning out decades' worth of papers during the spring?

- D's hypothesis is that N either had approved of a specification, but that X then had done something that differed markedly from what examiner N approved of, or, that N in an episode of temporary confusion (?) had approved of a specification that actually was in line with the thesis later submitted.

- D talks to the thesis administrator T - who of course has no recollection of this specific student. Furthermore, T was pretty new to his job at the time as he took over the responsibility for these tasks last summer, a few months before student X turned up.

∑ We don't yet know what has happened but it's a mess.

Act 3 - Comprehension

- After having talked to all involved parties at least once and sometimes twice (including student X twice), D now knows what has happened:

- Student X studies for a master's degree in media management -- as part of that degree, all students take a compulsory course in media management at SSE -- students also have the possibility to take other courses at SSE and X studied a course on "Purchasing" -- X got hold of the thesis topic through that course and pursued it.

- While each step follows from the previous step, the whole process is short-circuited when X takes his thesis back to KTH. In this old-new context, the thesis is evaluated and judged as a thesis about media technology/media management and judged based on criteria from that area. It might (or might not) be a great thesis about purchasing and outsourcing but that is unfortunately besides the point at this point...

- It turns out that student X has actually never gotten his proposal accepted by examiner N or anyone else at KTH. No-one at KTH has seen his proposal and no-one at KTH has had any idea about what X has been up to during the previous 6 months. For all we knew, he might have been on vacation or something.

∑ This is a mess.

Act 4 - The innocent

- These are the cast of persons that have been involved but are in no way to blame for this royal screw-up:

- Formally co-, but de-facto main supervisor H has had no idea at any time during the process that student X is not an ordinary SSE student, but rather a student at KTH studying for a master's degree in media management.

- Examiner N has never seen and never approved of a thesis specification and neither has examiners J & D.

∑ Innocent by-standers

Act 4 - The blameworthy

- I know our webpages aren't very easy to find your way around, but here is the webpage where it states how the process of starting to write a thesis should look like.

- It is difficult to know what information thesis administrator T gave to student X back in October but it is clear that it wasn't sufficient. At that time T actually saw (and wrote down) the proposed title of the thesis (already then a strange title from a media technology/media management perspective). He could have reacted, but it's not really up to him to judge the suitability of a proposed thesis title - that is done by the examiner when the student sends in the specification.

- Thesis supervisor B at Handels claims to not know that it was a KTH thesis (see above), but he has willingly signed a paper with a large KTH logotype on it, stating that "he approves" of the thesis specification. However, not being a teacher at KTH, it seems strange that he willingly and without any questions signs important papers from another university...? Furthermore, any supervisor from "elsewhere" (for example from SSE) need to themselves be approved by us at KTH (together with the thesis specification) before they are allowed to supervise our students!

- Student X claims to believe that the thesis administrator's signature that he fulfilled the prerequisites, together with the SSE supervisor's signature that he approved of the specification made him believe that everything was ok with his thesis. Student X has obviously worked very hard with his thesis, but has unfortunately at no point stopped and asked himself if his thesis topic is suitable for fulfilling the requirements for his degree in media management. He has not read official information about the thesis course he is taking or about writing theses in general at CSC/Media technology. He has furthermore not compared or talked (enough) with friends of his who also study for a masters in media management etc. but rather (probably unknowingly) gone out on a limb.

∑ These persons need to get their act together

That's about it, or at least the main events. We now know what has happened and some about how to improve the chances of preventing this from ever happening again. What we don't know is what to do about this thesis right here and right now. Since I (examiner D) am not a regular examiner in the course, I have handed over the matter to someone else who has more authority to take a decision (but who is probably equally confused about what to do at this point). I'm thankful it's not on my table any longer, but I am still very interested in how this all will turn out in the end.

.

Program-integrating course 2010/2011 summary

.

This will be a relatively long and rambling post and I might go back and edit it later. It is equally much a blog post that might be of interest for students who take the course (and who would like to hear the teacher's perspective) or for the nine teachers in the course as it is a summary of the course that I might go back to later and review.

Formal information & basic structure of the course

-------------------------------------------------------------

The program-integrating course [DM1578, programintegrerande kurs i medieteknik] is a 7-credit course. There are 250 or so students who take the course - all students who study for a degree in Media Technology are required to take it. The course runs for three years and we meet four times per year and discuss a pre-specified theme as well as student courses and experiences during the previous quarter. The students are divided into 36 groups with half a dozen of students in each group and we try to mix first-, second- and third-year students. Each teacher in the course lead four groups of students each and the teachers report back to me. Students get 3 credits after the first year and 2 credits after each of the following two years.

Development of the course over time

--------------------------------------------

This course sounded wonderful on paper when the whole program started, but quickly ran into problems and the program filled up with students. How do you fill a course such as this with content? How do you run it practically? I gave the course for a few years before my colleague Björn took it over three years ago. He changed the structure of the course (most probably for the better) according to what I wrote above. As he was away, I took over the course during the 2010/2011 semester.

Basic challenges of the course

-----------------------------------

The huge challenge here is to practically administer the course. It is difficult, to say the least, to keep track of 250 students that meet four times in a year. On top of that we have students that are on leave, that study abroad for a semester or two, that miss one out of four seminars (or two) for reasons that are sometimes better and sometimes worse (forgot, overslept, trip abroad etc.). One of the problems when I took over the course was that many students had, for some reason, not gotten last year's credit for the course. It is difficult for both me and for the students to know exactly why the didn't get their credits. There is probably a reason (besides administrative SNAFUs), but no-one remembers and these things are probably not documented in a way that easy to find and understand. It is detective work to find out why someone didn't get their credits. It is another task to gauge the seriousness of the lapse and to think of what kind of extra task needs to be done in order to get the credits. It takes time and it is boring.

The main contributions on my behalf during this year

----------------------------------------------------------------

- Better administrative routines for the teachers to document and report on students' performance throughout the academic year and the seminars.

- We had a lunch for the teachers in the course during the autumn and another one during the spring. This gave us the chance to discuss the course, routines and how to improve it and was great. Course development on the cheap (the only cost was the lunch itself).

- Better routines and rules for what happens when students miss smaller or larger parts of the course. Students collect "bad karma" when they fail to do what they are supposed to. Bad karma accrue and results in larger or smaller tasks to be completed at the end of the year. Better documentation of status of students who don't complete this year's course so as to better track them later.

The main things that I have a bad conscience for not having done (yet)

-----------------------------------------------------------------------------------

- There are students who completed this year's course, but now get in touch and tell me they didn't complete last year's course. It is a hassle to deal with the these students (takes time and requires boring detective work) and I have a backlog of cases that I have not fixed yet.

The main thing that will change in next year's course are:

- A theme will be chosen and the four tasks for the whole academic year should be formulated already at the beginning of the term (August/September) so that they can be portioned out later rather than hang above me as a dark cloud. It happened that the students got instructions for a seminar relatively late this year and that is unnecessary and could be fixed relatively easy.

- Even better administrative routines and instructions for the teachers. Students will not be allowed to jump to another teacher's group because the allotted time for a seminar did not fit the student. It so often happens that this information is lost and this results in an added administrative burden for me and the teachers (and the students) in an already hard-to-administer-course.

This is the course where students evaluate (discuss) all other course they take, but we don't spread around a course evaluation of this course itself.

Books I've read lately

.

I wrote pretty recently about my reading habits in general and I have also previously written about the work-related books I read during the autumn. I more or less read two non-fiction books per month of which one is work-related. Here is a summary of the books I've read during the spring:

The Swedish-language 2010 anthology "After the Pirate Bay" ["Efter the Pirate Bay"] contained no less than 19 contributions of which a bunch were written by persons I happen know (one contributor sits in the room next to mine). The contributors have a variety of backgrounds and while most are researchers (from a variety of disciplines), some have other backgrounds (for example a few journalists, a member of the parliament, an entrepreneur). Almost half of the researchers are ph.d. students (i.e. junior researchers probably in the 20's or perhaps early 30's). They write about a variety of issues and the book is divided into the three different parts; "technology", "pirates" and "politics". The book was naturally a little uneven, but it was in general a good read and one of the texts in fact gave me an idea I hope to be able to develop in a shorter text of my own. My main take-away from the book is how totally out of synch current copyright/IP laws are with the technology of our times, and, with common sense! (Not that I didn't know it before...) Although the book (of course) is available for free online, it is inexpensive enough (69 SEK) for anyone who wants to read it to order the paper version.

Several in-your-face examples of just how out of touch and absurd our laws for regulating Intellectual Property (IP) are were glaringly obvious while reading Lawrence Lessig's "Free culture: The nature and future of creativity" (2003). I notice that the more recent edition of the book has another subtitle: "How big media uses technology and the law to lock down culture and control creativity". That just about summarizes it all. Lessig (whose earlier book "Code: And other laws of cyberspace" impressed me a lot) sounds like a lonely sane voice in a desert of lobbyist-fed propaganda from industry dinosaur-titans trying their best to obstacles in the way of of "creative destruction" making the process short with their previous-century industrial age business models. By looking backwards and clinging to the past, organizations such as the Recording Industry Association of America (RIAA) and the Motion Picture Association of America (MPAA) squarely stand in the way of new, better business models that are more appropriate for the time we live in. Unfortunately these previous-century dinosaurs command hefty purses and thus do their best to strangle common-sense attempts to free the positively vast amounts of culture that has little or no economic value in the marketplace (because it is old or of interest to the few rather than to many etc.), but that in the age of affordable networked computers is of vast value to of our societies and to our culture. Even though Lessig is not alone, he is one of the best in formulating his critique against the current state of affairs. Lessig is an astute observer and a great writer - but I can't keep up with him as he has written a new book every second year during the previous decade.

Paul Edwards book "The closed world: Computers and the politics of discourse in cold war America" (1996) is a book I have owned for more than a decade and I have in fact started to read it once before but got stuck 30 or so pages into the book. Although the book is a little on the heavy side to read it is extremely well-researched and manages to convey a picture of pioneering work in the computer sciences that is radically different from most of what I have read before/elsewhere. Edwards ties the development of digital computers (and cognitive sciences) tightly to American cold war goals and mindsets and military dreams of creating cyborg human-machine interfaces and war machines - or even getting rid of the slow, unreliable human in that loop! There are many things in the book that "rewire" the standard history of computers, software, human-computer interaction, artificial intelligence (AI) and cognitive sciences, for example the vast (often non-directed) support from the military for basic research in computer science in general and AI i particular during its first 20 years as an academic field. The final - almost 50 pages long - chapter analyzes the computer in popular culture (with an emphasis on films) from the 1960's to the 90's (Dr. Strangelove, 2001, Tron, Star Trek, War Games, Star Wars, Terminator etc.). The analysis livens up and ties back to and applies Edward's previous analysis of the much "heavier" and "drier" material, and it is brilliant!

The last book I read during the spring was Richard Sennett's "Flesh and stone: The body and the city in western civilization" (1994). This is another book that I have owned for quite some time and while I have read other, later books by Sennett with pleasure I have to say that this book was both a little heavier as well as a little further away from my academic/research interests that I thought before I started to read it. The book treats the relationship between the human body and the city in a variety of ways; how humans move, adapt to and conduct their affairs in relation to the design of the city and city life, as well as how the city itself is a product of culturally bound conceptions about humans, their relationship to others as well as their religion, cosmology and world view. The featured cities are (primarily) the ancient greek city-state Athens, Rome, medieval Paris and Venice, industrial-age Paris and London and modern New York. While I found the book moderately interesting, it was not the easiest read and it is doubtful I will have any practical use for it in terms of my (wide-but-finite) research interests.

Have you read any of these books (or would you like to)? What is your opinion about them?

.

On the fallible nature of tests and testing

.

My previous blog posts/rants about top-down vs bottom-up approaches to improve education and the paradox of planning is here complemented with some thought-provoking quotes about tests from Gerald Weinberg's 1971 book "The psychology of computer programming". I have the 1998 Silver Anniversary edition, the quotes below are harvested from pages 154-155. I have commented these quotes so as to make the connections to issues raised in the preceding blog posts more explicit.

"A number of firms have used the Strong test for selecting programmers [...] making a judgement that a programmer is "like" a mathematician, engineer, writer, or what have you. Since the basis for these judgements is pure speculation, such selection procedures could have been equaled by throwing dice. Throwing dice, however, does not have the right sound to it. A personnel manager can say, "We use the Strong Vocational Interest Blank to help us select programmers." This will certainly impress his manger more than saying, "We throw dice to help us select programmers."

If you don't really really know what you are looking for or what you are measuring, throwing dice might produce comparable results. Investment advice by monkeys throwing darts have been proved to often (around 50% of the time) be just as good as the advice of the average "qualified" investment managers. If you really really don't know what you are evaluating (or more generally, doing), throwing dice might be an attractive low-effort low-cost alternative). :-)

"Even assuming that the profiles are available [...] do they really reflect what we want? After all, these profiles are obtained by testing people already in the profession, not the people we would necessarily want to be in it if we knew what we wanted [my emphasis]. In old, established, and stable professions, it may be valid to assume that people in the profession are, by and large, the ones who should be in it - even though they might have been steered there by [...] a self-fulfilling prophecy"

Think less of "people" and "professions" and more about "educational programs" and you notice the conserving power inherent in the preoccupation of evaluating what is, and what has been. How do you develop a world-class innovative university education? By evaluating what has worked fine before/elsewhere/up to now, or by taking stock of the future and setting out in a new direction - even, or especially if that turns out to be in another direction compared to merely extrapolating from the past? I presume "best standards" approaches are fine if you have your goals set for mediocracy or slightly-above-average, but in order to do better than that you really do need to think for yourself (yourselves) and look forward more often than you (anxiously) look backwards over your shoulder.

"Essentially all psychological tests [...] assume that the psychologists who made the tests are smarter than the people who take them. Indeed [...] people who are attracted to psychological testing as a profession [...probably...] hold themselves to be smarter than other people. Perhaps it could not be otherwise, for how would they get evidence to the contrary? [...] In a way, a personality test is an intelligence test - a matching of wits with the person who made the test."

If a monkey in the jungle stumbled upon an iPad or or some such high-tech gadget, he would probably think it is a very stupid, or silly object - and how could it be otherwise? Do the people who create a "test" such as the Education Assessment Exercise, as well as those who buy in and carry it out, as well as the those who analyze the results and those who act on these results "by necessity" think that they are smarter than the people who "get the short end of the stick" and whose only role is to merely provide them with information? (Yep! It is an unequal relationship.) Will they "by necessity" think that their conclusions about "what is to be done" is of a higher caliber than the opinions of the flesh-and-blood university teachers who by filling out these tedious forms provide them with truthful (?) information on which they are to act?

Instead of filling out forms and providing others with information (and putting our hope in wise decisions made by some nebulous "them" residing elsewhere), a better way to improve an education might be to create the space-time for those who actually teach to regularly meet and discuss issues that they find necessary, interesting or problematic in their day-to-day, month-to-month or year-to-year activities! To "create time" means (for example) to protect the time of the faculty at a university from incursions by others - including centrally initiated requests for this-and-that or (some) students' extravagant expectations of getting personal answers by email within a day to any and every question they might pose (even though the answers might already have been provided at the introductory lecture and even though many of their friends might know the answer). Perhaps a central person (a process leader or mediator) would need to do no more than to put his ear to the ground and listen to the concerns of those who are enmeshed in the day-to-day operations and then raise some of these issues into points that are to be discussed among colleagues?

"As we know, applicants for programming jobs are likely to be a rather clever bunch, so we can assume that a great deal of "cheating" will take place if they are giveen [personality] tests. But that should not worry us, for if they cheat successfully, they are probably going to have a number of he critical personality traits we desire - adaptability to sense the direction of the test, ability to tolerate the stress of being examined by someone they don't know under assumptions they cannot challenge, assertiveness to do something about it, and sense of humor enough to enjoy the challenge."

Perhaps I (and other teachers) filled out the Education Assessment Exercise not with the goal of providing "accurate" and "truthful" information, but rather with the intention of making my own courses sound as good a possible, and in doing so attempting to aggrandize myself as a teacher...(job security, salary, image and social standing)? Perhaps I did not set out to do that, but it might still be difficult for me to provide information about the courses I teach without implicitly/unconsciously promoting myself...? (...How do I want to come across to my superiors and to other (unknown) people reading what I produce? ...What should I write in order to convey these impressions?)

Perhaps I (and other teachers) did fill out the Education Assessment Exercise with the goal of providing "accurate" and "truthful" information - but that information might still turn out to be of questionable value to someone who has no knowledge of the underlaying reality behind the information provided (all those beautifully formulations lovingly crafted to reveal some some facts and conditions and to sweep other under the carpet...)? Distant decision makers have the map, but relationship between the map and the underlaying reality is shaky. To the person with a hammer everything will look like a nail, and to the person with a map, every obstacle will seem easily skirted from the comfort of his armchair...

My conclusion is that the value of tests and testing in general is greatly exaggerated! The implications of this statement for what students do at universities are fundamental and far-reaching. It might become the topic of a future blog post.

.

The paradox of planning

.

How much time should be spent planning an event in relation to the time spent actually doing it? How much time should be spent planning a study or a software project in relation to the time spend actually doing that study or project? How much time should be spent evaluating a course in relation to the time spent actually teaching it?

When it comes to doing studies, most "junior researchers" (university students writing their theses) often spend way too little time planning something before they jump in both feet first. That is a pity, because by beforehand thinking over all the steps of a study and the process of performing one, you can "debug", find and solve problems before you even start - rather than halfway in. The same goes for writing a large, complex computer program - time spent planning is undervalued, and especially so by junior programmers.

But after you have written that computer program, how much time should be spent documenting it? A traditional problem in software engineering is that very little time is spent on documentation and the short answer would thus be "more". At the same time, there is a law of diminishing returns at play both in the case of documenting computer code or planning a study. Spending twice as much time does not necessarily lead to twice as good results, and spending ten or a hundred times as much definitely won't. At some point, it's time to say "enough".

Moving to the university setting, how much time should be spent planning and giving a course, and how much time should be spent administering and documenting that very same course? At some point, any endeavor will face a diminishing marginal utility of pouring more resources into it. My previous blog post concerned the diminishing marginal utility of trying to control and improve the quality of higher education by top-down approaches (such as through performing "Education Assessment Exercises"). If I give a course this year and I will give the same course next year, how much need is there really to document the course in excruciating detail? I can understand the need to do so if I were to hand of the course to someone else, but if I don't plan to do so, is it not enough that I do the work of thinking through the course, rather than writing lots of stuff up just for the sake of it?

My point is that those times that I have written up a course evaluation, nothing has come from it. Sure, someone may have spent a few minutes looking at it, but that is of no special use for me. So why should I be forced to write things up just for the sake of it?

On the other hand, I would be passionately willing to discuss the course in question with someone who has a sympathetic ear and who might offer suggestions for improvements, but this service has never been offered to me because someone else's time is a cost, but my time is free (in the eyes of my superiors). This is another example of a top-down approach (of ordering me to provide information that no-one ever reads, or at least never acts upon) and a bottom-up approach (of supporting me as a teacher with those things I perceive as problems in my day-to-day activities). It also unveils power relationships between me as a lowly and exchangeable provider of educational services, a cog in the machinery in an industrial model of education - and those who run the organization or the whole system of higher education in Sweden. In this model, students are the raw materials and teachers stand at a conveyor belt "treating" them in different ways until we in the end turn out "quality-assured" cookie-cutter engineers.

We never have enough money to provide hands-on support for teachers to help improve their courses, but we always have money to make sure that someone writes an email to remind me to fill out the course evaluation document and to perform an EAE assessment. Who reads the course evaluation document? How will the EAE assessment be used? Who knows? I for sure don't know! How come? Who knows?

.

Program integrating course

.

We have this course in our program where all students (250+) meet once every quarter of the year in order to reflect upon and discuss the courses of the previous period. It is for the most part a really good course, but it runs over three years and managing the administrative backend of the course is indeed a hairy affair. Unfortunately it is my task to it as I have been responsible for the course this year.

I can be very details-oriented, so I manage, but it is quite boring and it takes a lot of time. To track down the students who did not perform and to force them to an extra task (which some of them object to) is not the most fun of things to do either. Sometimes I have to invoke detective skills to try to make sense of things that happened half a year ago or more. But someone has to do it.

I've now spent several days just summing up the results of four seminars during the past academic year, with hundreds of students, divided into 36 groups, led by 9 teachers, all of whom report to me. The quality of the documentation I get from the teachers vary and so errors do creep in and I then have to juggle students that are upset or angry. Sometimes the anger is justified (errors to happen) but oftentimes it is not.

Fortunately I'm (almost) finished now. Phew!

.