Research Methods

**Hugo** · 05-05-2009

If you need help on Research Methods I may be able to offer some help though I cannot promise to read through you project or dissertation. Here is a sample project outline that might be used at almost any level.

Basic Chapters - these are the usual chapters to find in a whole project. You can add appendices as necessary but here I just show the ones which are almost always required.

Chapter 1 - Introduction and problem outline
Chapter 2 - Literature Review
Chapter 3 - Research Design
Chapter 4 - Presentation of data and generation of results
Chapter 5 - Evaluation of outcome and practice
Chapter 6 - Conclusions and Generalizations

Appendices – Specification, schedule, Glossary, References list and Bibliography, primary data collection/set. Other items that might be included in an appendix are: Inclusions (copies any relevant documents), Sample Questionnaires, Summary interview transcripts, Details Evaluation scripts, Requirement catalogues, etc

I might start here be asking a question: so what is your defintion of all the following: a project might generate an outcome (a model, a plan, a description etc) but is that the same as the conclusions?

**Hugo** · 01-25-2010

Here are some notes on Qualitative Precessing methods - there are quote a lot of them so I will deal with it over a few posts. In this post, an introductory and very important note and then two methods with several more to follow.

Qualitative Data Processing Mechanisms
There are four sources of data that you might want to process: text, audio, video and images. The principles in each case are similar but the methods will differ although it is possible to buy software that will help you do all 4 of these such as HyperRESEARCH™ Downloads. In this note only text processing is covered. Here I only outline methods for text processing but the principles are much the same for other types of data.

For example, when looking at text, say as interview transcripts you might look for what are called outliers (unusual or odd opinions) or common threads and in a similar way if you were looking at video footage you could look for unusual movements or scenes and common features between scenes. Be aware that processing video and audio is very time consuming because it becomes very tedious if you continually want to play back little sections or you might not be quite able to make out what is being shown or said. On the other hand, watching/listening is very good for picking up emotional clues because they can often be seen in body language or heard in the voice. Alternatively, there is software that will transcribe voice data to words (but it’s not always very accurate) and also software that can clean up a bad recordings. However, in both cases it is normally best to work out what you are looking for and then get some help so that one person can look for features A, B and C and another can look for features X, Y and Z or run through recordings noting important elements and then good back to them later.

Many projects deal with qualitative data and this note just outlines the sort of things you can do at the pre-processing stage in your research design. What I am doing here is a kind precursor to template analysis and you may like to research that term yourself. There is also a related technique called content analysis and it is much the same as far as ideas are concerned but differs in the detailed ways they are used.

Introduction
It is very important to keep notes of interviews, observations or as you read through documents and make sure they are structured and accurate otherwise you will find your biases coming out in the results – that is you will interpret what you find the way you would like it to be. As you know there are two phases of processing data in research. The first one is all about assembling your primary data collection and the second phase is to process the collection into the intended outcome. Don’t fall into the trap of collecting data and thinking that is all you have to do or ignoring it as far as getting an outcome is concerned; that can only have one result – fail.

PLEASE remember the processing activities suggested here are not deigned to generate a project outcome - these are pre-processing stages to analyse and organise the data ready for you to construct your project outcome. If all you do for example is apply several of these activities to your data then normally that well NOT be regarded as generating your project outcome. For example, suppose I have a set of interview transcripts and a set of observation notes all collected in an attempt to generate as my project outcome a best practice portfolio on office management development. So I might pre-process this data (transcripts and observation notes) to get: common threads, outliers and labelling but obviously those three lists cannot possibly be regarded as my best practice portfolio but they are a necessary step toward me generating it from them.

Qualitative Processing Methods
The ideas listed below are commonly used to look at qualitative data. I am not suggesting you use all of them but usually as the data emerges from your collection process you will start to get a feel for which ones might be the most useful. Commonly, people use a spreadsheet, Word or Database to deal with all this although it can also be done by hand. My preferences would always be a database because of the potential for indexing, searching or linking it to other data sets or even links within itself.

Common Threads - Are there common response threads running through your interviews, observations or documents as these indicate a shared understanding in your sample and might be useful in formulating the project outcome.

How to do it - use a tabular method to collect this evidence together by listing the themes and counting occurrences and variations.

Outliers - It is often useful to look for extreme or unusual opinion or events and they might point to serious problems in the situation you are investigating. These indicate that the understanding is not shared and often mean that further investigation and may lead to very useful insights.

How to Do it – use the same tabular record used for common threads but look for items on it where the opinion is different, unusual or extreme with very little agreement with any other sources with virtually no commonality.

Word Frequency - You can construct a concordance and look at how frequently words are used and of course it also gives you the situation vocabulary.

How to do it – use a software tool to generate the list of all words used and their frequencies. In fact it may be quite interesting to see how wide the vocabulary is as this might give you another “handle” on the problem theme. I would not recommend you try to do this manually unless the transcript or documents are of a very limited nature. Unfortunately, if the documents are not in an electronic form (unusual these days) then a manual process is the only one possible.

**Hugo** · 01-27-2010

Here is my second note on Qualitative Processing methods

Meaning - Make sure you know the meaning of any words used and what kinds of words are used: descriptive, explanatory, critical and so on.

How to do it – using a constructed concordance or glossary to list all the words used and then for the important ones or ones where you do not understand the meaning write a definition.

Semantics – this is just an extension of “meaning” but here you are trying to ensure that you understand what has been said not just know what the words mean– so one might look at phrases or sentences for example. This needs a lot of care because it is all too easy to see a meaning that you would like to see and not the one that is actually part of the data. It is unfortunately all too common for students to write what they think is a simplification in their own words or substitute an accepted situation word or words for one of their own and this can often turn out to be disastrous.

How to do it – it is hard to find an absolutely secure process here because a certain amount of domain expertise is always needed. However, a reasonable plan is to look for and write down the key ideas in the phrase or sentence because if you have the key idea then in practice one understands what is being said. One almost always does this is conjunction with the tabulated themes because themes themselves might be an object, an activity or and idea.

Labelling - It is often interesting to look at how situation elements are labelled. For example, the software might be labelled as ‘useless’ or ‘difficult’ or ‘perfect’ by users or the managers labelled as 'arrogant', 'unhelpful', ‘lazy’ or ‘sympathetic’ and ‘helpful’. When this happens it may indicate serious problems or a good environment in the situation.

How to do it – using a constructed concordance or glossary to list all the words used and then for what you regard as a label search for them in the list.

**Hugo** · 01-31-2010

Here is my 4th Posting on Qualitative Processing Methods

Structures - You can partition the answers into such things as: opinions, definitions, explanation, theories, concepts, methods, policies, governance, training, environment, attitudes and so on or any other categorisation that you can identify. This will help to ensure that you have a good understanding of the situation as seen from interviews or documents or other data collection mechanisms.

How to do it – use your tabulated list of themes and then add extra columns to code the structure that you have identified or want to use. Remember that an element may have more than one structure – something could be a definition but also be expressed in the form of an opinion.

Response Validity - Always ask is the response valid or relevant in that situation or can it be discarded. The reason you want to do it is because you don’t want to be encumbered by data that has no value and one is always looking to get the smallest valid data set.

How to do it – this is not easy to do and it requires good to very good domain knowledge. I would recommend you try to generate a few simple questions that you apply to the data and if they all give a yes answer then accept the data. This might be things like: “is it a common opinion”, “is it a fact”, “is it interesting or insightful” that sort of thing but don’t have too many questions else you will end up with no valid data!

Response Reliability - Always ask how reliable is the data that you find or are given - this is to do with how the interview or search was carried out and can you rely on it as being truthful.

How to do it - Essentially we ask would we get the same result if we did the interview/search again.

**researcher** · 02-03-2010

Hi Hugo,

Haven't been on here in a while as I have been busy with studies. I'm back with another question! By the way I passed my philosophy Alhamdulillah - chuffed!

I have a question, I'm hoping you can help - unfortunately, I missed the seminar and now feel a bit stuck.

What are latent variables and manifest variables? How do we identify them? I'm very confused... I'd be very grateful if you could perhaps give examples or scenarios of these.

Many many thanks,

By the way your posts of qual and qaunt analysis etc are very timely as I am currently completing modules on both!!

**researcher** · 02-03-2010

even if you could point me to a text or article that may deal with them I would be more than grateful

**Hugo** · 02-04-2010

Originally Posted by researcher

Haven't been on here in a while as I have been busy with studies. I'm back with another question! By the way I passed my philosophy Alhamdulillah - chuffed! I have a question, I'm hoping you can help - unfortunately, I missed the seminar and now feel a bit stuck. What are latent variables and manifest variables? How do we identify them? I'm very confused... I'd be very grateful if you could perhaps give examples or scenarios of these.By the way your posts of qual and qaunt analysis etc are very timely as I am currently completing modules on both!!

In some ways this is very simple but it can sound a bit odd. I don't have an example from psychology but here is one from software but I think you can follow it.

At a very simple level a variable like area or volume are latent variables because you cannot measure them directly but what you do is measure length, depth and so on an calculate them. Now, suppose I want to measure the QUALITY of a bit of software, well software is not something you can see or touch so I have to ask myself what do I mean. So there is no variable to use (like feet and inches) just as in the same way if you are measuring the quality of chair or table what exactly would you use? So I might do it by counting the number on lines of code, counting how many IF statements there are, counting how many functions etc. These are my 'manifest' variables because I can 'see' them and count or measure them. But it would be hard to say these measure quality or if they do it is indirectly.

So here I might define a standard program outline and then compare all other programs to it. If I do that I could then end up with a scatter plot where one point represents one program. Now suppose I did this for 60 programs and measure how far any given program is from the standard and that then that distance become my measure of quality. In fact this measure is called planar similarity and it is a latent variable because I CANNOT measure it directly and it only emerges once I have the basic data set.

Consider if you wanted to measure 'coolness' in people, are they 'cool' or not. Well I don't know what to do here so I decide to collect data such as: age, musical preference, religious leaning, etc. I am JUST being simplistic here but you could give a score for each of those variable between 1 and 10 (age becomes , child = 1, youth = 2 etc). Then we just add them up and divide by how many manifest variables there are and that becomes your measure of 'coolness' - can you see its a latent variable because it is not observable direct?

There are much more sophisticated statistical ways of doing this and you might look at factor analysis or principle components but you cannot do it by hand you need a computer. You must understand to that it is very arbitrary as you can see from both my examples and some regard factor analysis as very unsatisfactory but principles components is much safer but in both cases you must not be taken in by thinking you have uncovered something like gravity.

**Hugo** · 02-05-2010

Here is my last but one note on qualitative processing methods.

Significance - Can you identify items that are clearly significant in this situation – significance here means that the response is representative of something genuine.

How to do it – just look at the tabulated frequencies for the main themes, labels and significant words used. One cannot be certain from just frequency that it is significant so one must also weight it up in you mind against your research question and whether you can do something with the data. For example, to be flippant one might get a common theme emerging that there is not enough car park space and things like that it is almost certain you can do nothing about.

Generalise - Is there anything that leads you to make generalisations.

How to do it – essentially what one does is look at themes, labels, outliers and knowing these have emerged from a sample we now try say what it might mean for the whole company - can it apply to the whole company, is there some important element in this theme that has a much wider implications, is there a principles that can be established, can I construct a theory and so on.

**Hugo** · 02-13-2010

This is my last post with regard to processing qualitative data and it is also the most complicated and difficult and cannot be done by hand. I offer one example from technology and one from Psychology.

Statistical Processing as Often Applied with Nominal Data
Qualitative data can be used to extract variables but almost always the variables or either ordinal or nominal and in general that means they are arbitrary and you have in effects invented them. For example, in a qualitative study of office personnel you might have variables called: trustworthiness, honesty, coolness, reliability, personality and so on where it is obvious we do not have universally agreed scale on which to measure these qualities. In passing I should say here that you should make all such variable go the same way and in general that should be a positive direction though each study must carefully consider what it needs. So it is better to look for honesty (along some scale) that dishonesty as you will for example have enormous trouble if you try collecting data how dishonest people are.

Now in studies like this it is very common to find that we cannot measure the variable we want directly, either because it is a complex issue or we have no idea what the scale is or because it is just inaccessible. The literature in this context will often talk about ‘manifest’ variables, meaning variable we can record directly and ‘latent’ variables which we cannot. At a very simple level a variable like area or volume are latent variables because you cannot measure them directly but what you do is measure length, breadth and height which are manifest variables because we can obtain them directly by observation. Once we have length, breadth and height we can calculate the surface area and volume which are latent variables because we cannot get at them by direct observation. Here are two more realistic research examples:

Software Engineering
Suppose I want to measure the QUALITY of a software program; well software is not something you can see or touch so I have to ask myself what I mean by quality. So there is no obvious variable to use (like feet and inches) just as in the same way if you are measuring the quality of chair or table what exactly would you use? So I might try to do it indirectly by counting the number on lines of code, counting how many IF statements there are, counting how many functions, counting the density of function calls etc. These are my 'manifest' variables because I can 'see' them and from a copy of the program code I can count or loosely measure them. But it would be hard to say these individual counts measure quality or if they do it is indirectly but I can argue that some aggregate of them does, the problem is what aggregate.

One possible way is to define a standard program outline and then compare all other programs to it. If I do that I could then end up with a scatter plot where one point, some aggregate of all my counts, which represents one program. Now suppose I did this for 60 programs and measure how far any given program is from the standard and that distance becomes my measure of quality. In fact this measure is called planar similarity and it is a latent variable because I CANNOT measure it directly and it only emerges once I have the basic data set to which I apply a statistical process called ‘principle components.’

Psychology
Consider if you wanted to measure 'coolness' in people, are they 'cool' or not so this is a complex idea and let us say I decide I cannot get at it directory. Therefore, I might begin by making a list of emotions both positive and negative, but remember such a list is arbitrary and others may not agree with you either about the emotions or about whether they are negative or positive. So don’t fall into the trap of think it’s all obvious.

Positive - happiness. joy, bliss, delight, amusement, contentment, pride, thrill, gratification, satisfaction, acceptance, love, friendliness, whimsy, euphoria, trust, kindness, affinity, devotion, surprise, wonder and amazement.
Negative - anger, outrage, vexation exasperation, indignation, acrimony, animosity, hostility, sorrow, gloom, melancholy, self-pity, low self-esteem, loneliness, despair, anxiety, apprehension, edginess, dread, panic, shock, contempt, scorn, distaste, revulsion, guilt, embarrassment, remorse, chagrin, humiliation, regret, contrition, no skills.

In a study I could choose say 5 positives and 5 negatives that to me might imply ‘coolness’ or I could just choose positives and these choices will depended on you and how you see this thing called coolness but they will never be more than arbitrary at this stage or in the future. Sop suppose I choose:

Positive - amusement, contentment, thrill, gratification and whimsy.
Negative - melancholy, contempt, chagrin, humiliation and regret.

Let us suppose I use a questionnaire and a 5 point bi-polar scale of 1 to 5 where 1 = strongly disagree to 5 = strongly agree, so I might word each question carefully to match this scale and set of variables in the form: Is life for you amusing?, Are you content with who you are?, Is melancholy something you avoid?, etc. Being simplistic here I could just add up the values of each manifest variable and divide by 10 and that average becomes my measure of 'coolness' in that office or that person - can you see it’s a latent variable because it is not observable directly? There are much more sophisticated statistical ways of doing this and you might look at factor analysis or principle components but you cannot do those by hand you need a computer. You must understand to that it is very arbitrary as you can see from both my examples and some regard factor analysis as very unsatisfactory but principles components is much safer but in both cases you must not be taken in by thinking you have uncovered something like gravity.

For example, if I were using principle components I could calculate how near (or you can say how far) each point is from every other point. If I then took those calculated values and turn them into a cluster plot and we might find just one cluster or several and each cluster might be thought of as representing a level of coolness. Here your latent variable, coolness, might just be regarded as a position on a chart so we then might say points towards the upper left hand quadrant are the coolest and those towards the lower right hand quadrant are un-cool.

It is also possible to look for outliers, points that don’t seem to fall in any cluster and so in your final processing you could look at say short biographies of everyone in the study and see if you can match these biographies profiles to the clusters or to outliers and hence try to say what coolness is but it will only ever be subjective but it is not really possible to go further as cultural things are never going to be stable over long periods.

**Hugo** · 02-18-2010

Taking a Random Sample
Often in research or projects one has to define a random sample so here is a simple example of how that is done.

Define the population - start by defining the population and usually this is done by setting criteria and then estimating how many people or things that might be. For example, suppose I want to sample all second year students on computer courses who study mathematics and I estimate this to be 100 although in this case I might be able to get students records to tell me exactly how many there are but there are always drop outs so in a way it is an estimate.

Define a Sample Frame - this just means a list of some kind from which you will actually choose your sample points. In this case let us suppose I can get hold of class lists and it might look as follows where I number each student from 1 to 100 (in this case). In fact as long as you number this list systematically at this point you can start and finish anywhere - so you could number the frame from 200 to 299 or 87 to 186 and so on.

001 John Ashman
002 Paul Brigham
..
095 Janet Brown
...
100 Anthony Zaccari

Decide or Calculate a Sample Size - there are many ways to do this but just for example purposes let us say that I want to choose randomly 20 students for my research study out of the 100 I have available in my sample frame.

Generate Random Number - now I must generate 20 different random numbers between 1 and 100 (or between whatever systematic numbering for the frame you used). If the generator gives you the same number more than once just discard them. Here I use the Iphone app AppBox Pro (a tool box of apps) and use the one called Random by telling it my range (1 to 100) and then pressing a click icon (or you can shake the phone) and it give me the list:

68, 33, 61, 89, 17, 24, 73, 80, 01, 50, 85, 92, 60, 95, 37, 72, 79, 21, 28, 11

If you which and convenience write them out in order:

01, 11, 17, 21, 24, 28, 33, 37, 50, 60, 61, 68, 72, 73, 79, 89, 85, 89, 92, 95

DO NOT be tempted to tamper with this list so stupidly say to yourself things like 72 and 73 cannot be 'right'. You MUST trust that the Iphone app has done its job and given you a random list. I warn you, more problems that you can imagine occur when people try to second guess these sophisticated random number generators - just trust them.

Select the Sample - now go through you sample frame selecting the students based on these random numbers. Again BE WARNED do NOT try to second guess no matter how tempted you are. Thus we end up with our sample of 20 students.

001 John Ashman
011 Paul Aldridge
017 Victor Litchmore
021 Gaetan Madhvani
..
095 Janet Brown

These are now your selected 20 sample points. If you wish you can select a few more in case some refuse to take part

**nurgee** · 02-19-2010

Assalammualaikum,
I'm carrying out this research that requires me to study the documentary films. I'm wondering whether anyone has done this before and care to share their experiences or opinion.

**Hugo** · 02-23-2010

I thought it might be useful if I posted some thoughts now on quantitative processing methods; where you have numerical data of some kind and you might be looking for predictions from it.

Warning - just because you “can” do the things that follow, do not be seduced into thinking it will automatically be meaningful or useful. That is you MUST think through what you want to do and when you have done it see what it means. From an assessors point of view the worst thing to see in student work is mindlessness and in this case just working out a pile of statistics - which ANYONE can do, and leaving it to the reader to figure out what it all might mean

Summarising Statistics
There is a wide range of descriptive statistics available to you and of these the most commons are described below and in CN.003 Statistics Notes but you may need to do further searching if one or more of the method appeals to you.

Cross Tabulations – here one draws up a matrix with typical respondent classification data as the rows and the various dimensions you are looking at as the columns. This is quite useful for summarising data and also is quite helpful for various kinds of statistical measures such as chi-squared.

Bar and Pie Charts - here one draws a bar or segment for each variable against the frequency of occurrence. Often these are used with colour to enhance their illustrative purpose.

Cumulative Charts or Ogives – these are just simple charts of a main variable against its cumulative frequency of occurrence. These charts are often used to estimate the proportion of observations above, below or between specific values. For example, 50% of subjects respond to the stimulus within 52 seconds. This value, called the median, is one of many averages that might be used to describe a set of data.
Dot Plots – these just give a one dimensional picture of the distribution of values.

Line graphs - these are just a simple plot of one variable against another. The implication when you do this is that there is a relationship between the variables - that is, knowing one of them will allow you to calculate the other. We will look at this later under regression and correlation. In simple terms this might allow us to arrive at a formula for connecting the variables. That formula might be linear or it may be more complex, but if we have a formula it is obviously very useful in predication if we can trust it. But be aware that these formulas are all about nominal laws they are not natural laws that are always true everywhere.

Box and Whisker Plots - A diagrammatic representation of quartiles together with the largest and smallest values, illustrating the distribution of measurements over the observed range.

Scatter Plots – a two dimensional representation of the data. Most often these kinds of plots are associated with correlation studies. It is also possible to use multi-dimensional scaling to effectively represent several variables in two dimensions.

**Hugo** · 03-02-2010

Statistical Measures
There are a large numbers of these and I am going to list the most common in a moment, however it is important you understand where and when these apply, how they work, how best to practically use them and how to interpret the actual statistic generated – they must not be used without this knowledge and either you must gain such knowledge or work with an expert before attempting to use or interpret them.

Average - this is just the sum of the numbers in the list divided by a count of the numbers in the list.

Range - the coverage over the largest and smallest values - so if the largest was 154 and the smallest was 80 then the range is 154 - 80 - 1 = 75

Mode - this is just the most commonly occurring value

Median - this is the value above and below which half of all the scores occur - if you like it is the very centre of the distribution. The median is almost always only used for frequency distributions.

Standard Deviation – this is a sophisticated measure of the spread of data about the mean or sample average and is of huge importance in statistical theory and practice.

Regression – this is about computing a line that best represents the relationship between variables – in simple terms you can find a formula that links the two or more variables. In most cases we just deal with linear regression – that is fitting a straight line to best fit the data. Once we have the line we can predict one variable from the other. Try to remember that because you can predict one variable from another it does not mean that the one necessarily causes the other. For example I can fine a line of best fit between shoe size and IQ but I would be very unwise to conclude these two are related. The most common method to find a best line is to use a technique called “least squares” although you can also do it by hand with a ruler.

Correlation - The relationship between two variables often carries with it the implication that one causes the other. In simple terms it tells us how accurate the formula found from regression analysis is. However, there are often many variables involved and some of them you may not be aware of – for example in the spring the number of birds increases and the grass get greener but you also know the length of the day increases so you are unlikely to conclude that that the extra green grass cause more birds to appear. However, it is not always as obvious as this and it is very easy to think you have a causal relationship without realizing there is a third or several other variables involved. So having a high correlation coefficient is a necessary condition for a causal relationship to exist but it is not sufficient. The most common correlation coefficient is that attributed to Pearson and commonly just called ‘r’.

Confidence Limits – these allow you to specify how certain you are about the accuracy of your results. To illustrate this take a simple example. Suppose we are trying to guess the weight of a man simply by looking at him. We cannot place him on a set of scales! We might say that he weighs about 100kg. When you say “about 100 kilos” are you confident that he weighs exactly 100 kilos? Probably not, so you might say “well I don’t know his exact weight, but I am confident that he weighs between 95 and 105 kilos. These are your confidence limits and the interval they give is called a confidence interval. Of course, you would probably be more confident about saying “His weight lies between 80 and 120 kilos” and even more so if you said “His weight lies between 70 and 130 kilos”. The bigger the interval the more confident you are in your answer. So how confident should you be? Using something called the normal distribution; it is (relatively) easy to calculate the confidence limits so that you can be 95% certain that, in our example, the man’s weight will lie in the appropriate interval. And 95% certainty is what we usually go for. Sometimes, if things are critical, you might go for 99% certainty, but this much rarer. In our example, we have given an interval that is equally above and below our initial guess of 100 kilos, this is called a two-tailed interval. There are times when we can say that the interval only makes sense for values above (or below but NOT both) our guessed value. In those cases we have what is called a one-tailed interval.

t-test – this is a common way to see if there is any significant differences between the averages found in two (or more samples) drawn from the same population. For example, on a questionnaire one might have two questions which refer to the same dimension and it would be reasonable to ask if the response to these two questions is significantly different.

Chi-squared - this is used with categorical data and is used as a “goodness of fit” test by using a contingency table – if you like it is a simple correlation test. That is, you start with a hypothesis and you see if the values you have obtained fit that hypothesis or not. For example you might set up a test to see if the colour of a car is more important to women or men. So you postulate there is a difference and then see if the test supports this. Occasionally the chi-squared is modified with Cramer’s V test but this is really only for normalisation purposes.

**Hugo** · 03-11-2010

Project – Evaluation
Considerable weight is given to the writing of an evaluation of how your project went and the results you obtained. It is expected to be in two parts - evaluation of practice and then your outcome:

Evaluate or Test your Outcome – implies mapping out what must be done to test the outcome when you finally get it but BEFORE it is used; it follows that project outcome evaluation is in most cases a paper exercise but there are two broad categories of evaluation and they are sufficient for most project outcomes and practices. In other words you ask will your outcome work, will it do what you intended. For example, suppose you generated a new model for internet sales then you must here say how you tested that model and whether it is likely to work. Two forms are possible:

Formative - implies that the evaluation is carried out before the full use of the outcome before full implementation. In general this is the normal situation with an MSc project.

Summative - implies that evaluation of the outcome is done after implementation and may use beta testing, comparative studies or consumer surveys or any number of other techniques.

Evaluate or Test your Practice – here students must say how they will reflect on the various process choices made; a plan for evaluating project practice must be done AFTER it has been carried out. Reflecting on practices will often uncover deep meaning about the nature, real purpose and intentions of you own deliberate actions and assumptions and these may prove uncomfortable but are a necessary learning device. For example, you will have made a design choice for a research method so here you have to reflect on that choice and see what went right and what went wrong so that lessons can be learned.

Finally, the evaluation is done BEFORE the project document is finalised by writing ones conclusions as a final task. It must be emphasised that writing your evaluation is NOT the same as writing conclusions although the one may inform the other. Conclusions are about generalising your outcome and to do that you need to think about your outcome, its evaluation coupled with your expert knowledge gained from the literature review. In both these aspects a thorough preparation from the literature is an essential step otherwise one simply does not have the requisite knowledge to be meaningful here.

**Hugo** · 03-15-2010

Project – Styles of Evaluation
As one might expect there are two major styles: quantitative and qualitative.

Quantitative - finding measurable counts or amounts or more complex ones based on rules (formula if you like)

Qualitative - finding un-measurable but observable elements; broadly.

Context - is the problem setting ‘better’ because of the use of your outcome?
Representative - are the observation of some element relevant and if so how?
Richness and Depth - the object is to know more about a few very central issues or features.
Ambiguity, Interpretation and Understanding - everyone sees the world differently and interprets what others say to gain understanding.

**Hugo** · 03-21-2010

Evaluation Principles and an Example
It is VITAL that you understand that evaluation takes place at a point in time AFTER you have collected the data and processed it into your intended outcome so your project is essentially over; implying that you CANNOT go back and correct or adjust anything and do it again but of course you can qualify or add riders to the outcome or its usage. Evaluation is also a difficult activity and will show the level of competence you have more clearly than almost any other section.

Evaluation is always in two sections and both are essential for serious reflection on complete project description write-up. Firstly it is usual to evaluate research practices because there may be implications which have a bearing on the outcome or is use. Secondly one evaluates the outcome itself to see if it has any value and if it is likely to be useful.

Evaluation Plan for reflecting on Project Practice – this is a plan and so the implication here is that when the work has been completed you take some time, you slow down, and look back at all you did in to eventually generate the outcome (literature review, data definition, data collection etc) and consider if there are any lessons to be learned so that the next time you do research it will be even better or were there any faults or errors that might have affected the outcome. This has to be honest and thoughtful not just a token gesture writing anything that sounds remotely like it might be useful. For example, on looking back you may feel the literature preparation was too shallow, interview questions were slightly biased, your attitude was unhelpful in data collection, the project plan was over-optimistic and so on. It is a not necessary to review every single thing you did but at a minimum readers would expect to see a considered evaluation of your project plan, literature review and your research design.

Evaluation plan for testing my Outcome – here you must carefully consider how you will check your outcome to see if it has value in the sense that it preserves or creates something good and it will be useful in bringing about the sort of change (policy, procedural, structural or attitudinal) that you hoped for. As a rule this is an entirely paper exercise, meaning you cannot actually put your outcome into use as a test as that would amount to foolishness because you are then using something before testing it.

The methods used are dependent on what your outcome is but commonly one can: map it to a suitable model, perform a functionality walk-through/story boarding of implied actions, use focus groups discussion, seminar sessions, comments from users, trials or simulation, ease of use etc. It is not expected that you use all of these but a selection that you feel best helps you do the overall evaluation. It is possible that your evaluation indicates that you need to modify or perhaps qualify in some way your outcome and that is acceptable.

Example
Reflecting on Project Practice
Firstly I will review the original project plan against my project log looking for discrepancies or errors and where these occur establish a problem cause. Secondly, I will consider the adequacy of my literature reviews and in particular look at sources, authors and coverage noting on reflection any gaps or serious omissions. Finally, I will evaluate my research plan and given the rather difficult sampling scheme planned here my focus will be on the supposed population, sample frame and sample and how tolerable they actually turned out to be. It seem also worthwhile to look with care on the activities in the processing scheme as at the design stage one could not be precise as to what that data might look like and so many changes might become necessary.

Evaluation plan for testing my Outcome
The intended outcome is to be an action protocol on dealing with phishing incidents and as such it is essentially it is based on policy decision and it is hoped that the new protocol will bring about practice and attitudinal changes in its users. As such it seems best to test such a protocol firstly on some sample phishing examples and then discuss it fully with interested parties in a seminar situation created to get critical feedback regarding its feasibility and effectiveness and also consider if its use it ethically acceptable.

**sully** · 03-25-2010

OMG! Hugo, where have you been all my life?!!!! Erm...ahem...how do you do? Nice erm piece of thread you got there. Would you be able to give me a short example for each chapter e.g. what is literature review? I'm doing my project on Computer networks and erm..I'm a bit thick. Can you explain to me how to do a dissertation in dummy language? I'd really appreciate it:-). Ta babes, you're fabulous! - I mean thank you kind sir!

**Hugo** · 03-26-2010

Originally Posted by sully

OMG! Hugo, where have you been all my life?!!!! Erm...ahem...how do you do? Nice erm piece of thread you got there. Would you be able to give me a short example for each chapter e.g. what is literature review? I'm doing my project on Computer networks and erm..I'm a bit thick. Can you explain to me how to do a dissertation in dummy language? I'd really appreciate it:-). Ta babes, you're fabulous! - I mean thank you kind sir!

I doubt that you are thick but you may be a bit lazy. In a short post its impossible to do what you as so I just offer what chapter 1 might look like

Chapter 1 – largely about scene setting and outlining the basic research elements thus, all the following must be covered although you do not have to use these sub-headings

1.1 Introduction with problem setting and client
1.2 Presenting problem, its causes and reason for its resolution
1.3 Overview of Research Plan covering: approach, style, brief study plan, primary data, outcome, actor and target and it is recommended you present them in this order as a series of connected sentences or as bulleted points. It is important that you refer to you project plan when you write this section.

Approach - inductive or deductive and you may present a hypothesis if it’s applicable
Style – qualitative or quantitative (recall that these refer to the type of outcome NOT the type of data)
Study plan – give the briefest of outlines as to what you will do
Primary Data – brief outline but make sure it’s understandable
Outcome – the final project product that will be used by the actors (report, review, model, plan, etc)
Actors – those who CAN and will use the outcome to eventually deploy IT assets to affect a solution

Target - the effects that would be observable if the outcome is used

1.4 Scope (what aspect is covered) and Scale (how many firms, people etc are involved). You may also include here any assumptions made or limitations on your study
1.5 Ethical Overview
1.6 Research Question: interrogative, outcome, actor, problem, spotlight, activity and target
1.7 Aim: activity, outcome, spotlight and target
1.8 Objectives: activity, spotlight, milestone (visible features) plus bounded and progressive (non-visible features)
1.9 Summary and link to next chapter

For item 1.4 you are trying to set limits on what you will do and hence limits on the applicability of the outcome so this needs careful thought. For example, I might set the scope as looking at eMarketing effectiveness and my scale is to do it with three different companies. If you wish you can add in this section a brief note on the methods you might use to show they are appropriate within your chosen scope and scale.

For 1.6 the order in which the features are written down will vary depending on the interrogative used so this aspect needs very careful thought. Items 1.7 and 1.8 may be combined into one section for convenience and the order of the required features when written down may vary as is best to ensure lucid wording.

**Hugo** · 03-31-2010

Project – Conclusions
It is easy to become confused here so you need to be aware:

Findings - these are your processed data, charts and stats. These are what you start with, they are NOT the project outcome or conclusions.

Outcome - this is what your create from your data. It might be a model, a plan, an explanation, a report, a protocols and any number of other useful artefacts. But the outcome is specific to your data.

Conclusions
Here one tries to go beyond the obvious and attempt to generalise on the topic area based on your findings/outcome. This is intrinsically difficult because no matter what your research approach was you are now going to look inductively into the future and say what you results might mean in a wider world that the one in which you collected the data.

The heart of the problem is how we can logically go from specific instances to reach general conclusions. How can we possibly know that what we have observed in our necessarily limited research on given objects and events be enough to enable us to figure out or derive their more general properties. That is, suppose you use your primary data to build a model of human/computer technology relationships. Well that is fine but that model was built using a tiny set from the possible data population so how logically can you get from there to making predication about its use in the wider world of you own company and elsewhere.

For example, you might have set your project in the UAE on the topic of modern communication technology and generated a position paper for your company. That will have been followed by an evaluation specific to your position paper and your company. However, in conclusions we must now ask "do any of the findings have general implications" and these might cover some or all of the following general ideas.

New meanings, originality, implications, new or modified principles, limitations, new or modified theorisations, indications of best practice, unanswered or new questions, lessons learned, indications of a need for further work, implication for law or standards, warnings or cautions, advice, caveats, values, ethics, factors or features including cultural ones, usage and user psychology and other things that might occur to you.

One might say here that some argue that the purpose of research is not to find facts but new and better questions and so one never comes to then end but is always moving forward. Mostly in the above one focused on your finding/outcome but it is also worth considering the methods you used to get them. Be careful here because this must NOT end up the same as evaluation because evaluation is project specific and here is it about making generalisations and that is often not easy to do. In summary to be able to write an adequate set of conclusions you must be competent in the topic area gained through a literature review and have a deep understanding of your own results.

Conclusion Strategy
It is not easy to be precise as to how to write conclusions that are generalisations of your observation and findings. However, one way is to start by using you detailed knowledge of the subject area based on your literature review headings to write topic area commentaries in an a priori manner; commonly this is focused on benefits and features. For example, you can look at international standards, government or organisational policies and do on and all these will give you comment area ideas.

Once you have a commentary you can take each paragraph in turn and re-write it in the light of your findings. Suppose I carried out a survey on the use of Skype as a modern day communication tool I might start off with this topic area commentary paragraph bellow (one of many of course).

Topic area a priori comment - the only real difference is that Skype can easily handle video calls and conferencing simultaneously. If these two services are the only differences, then subscribing to Skype might be impractical. If a task can be done by one tool, there is no need to use another gadget with the same purpose. Although, video streaming makes the conversation more intimate, spending the extra amount of time because of video may be non-viable. This does not mean that Skype is a failure among users as many respondents showed high degrees of interest and enthusiasm.

The above commentary is roughly written but that does not matter as it is just a means generating comment to merge with some the survey findings to become.

Final Project Document Version - In the survey finding one notices some let’s call it confusion over having a wide range of features of widely different facets all in one package: voice, video, chat, conferencing, file transfer, multiple calls and so on. Now, one might argue that many technologies such as simple email offer similar features so an unimaginative user might ask “so what do I gain, if a task can be done by one tool is there any need to concentrate them all into one box as it were? So on a larger scale business might just feel this is just another “gadget” to be managed and therefore not see its true potential. For example, video streaming in skype makes the conversation more intimate and immediate but does it do much more than that is the real question?

This does not mean that Skype is a failure among individual users, indeed, the fact that many respondents showed high degrees of interest and enthusiasm implies that Skype generates positive perceptions from many users and perhaps that is what is really needed; a sense of optimism and imagination to see how this new technology can be used for positive business and social purposes

**Se7ene** · 03-31-2010

I just got my mark back for Semester 1s Research Methods report

and I saw this thread. Its a sign

I'm definitely going to need to bookmark this thread.

**Hugo** · 04-09-2010

I thought I might now add some definitions and models as they can help in research. Some I will cover are well know other less so but all will need further effort to make them your own.

Policy Model
A policy is an expression of a prudent mechanism for controlling or limiting actions based on an underlying ethic as expressed in the Company/Organisation mission or perhaps more specifically as values meaning that the policy is intended to bring about some good; that which is worth having in the sense one want to get or keep or preserve them. So for example, in the University we have a policy for assessment and that controls and limits what departments may do. A policy is most often accompanied by a strategy to deal with various aspect of the policy. A good way to think about a policy is to see it as having five elements (APLOM):

A - Assumptions – Every policy will be underpinned by assumption of one sort or another about the workforce, the technology and so on

P - Principles and Values - are based on organisational values or on legislative or contractual elements

L - Links - to other organisational policies or other documentary sources. If you are not careful here you will find yourself overwriting or changing other policies that already exist in your organisation instead of referring to them.

O - Definitions - of the objects to be controlled

M - Monitor or Track - elements that set limits on what is permissible. This element will form the bulk of the policy definitions

Therefore the construction process is to set out the principles involved and make sure you are aware of any other related or relevant policies, legislative or contractual elements. Once the groundwork is done you can set about defining the object to be controlled and lastly set up how they are to be tracked and monitored.