Research Methods

I expect many of you are getting ready to go off to College or University and I thought you might like to hear some words of encouragement. They were made by Sir Ronald Pitts Crick who died July 2009 aged 92. You may not have heard of him but he was an eye consultant and almost single handedly he changed the attitude to glaucoma right across the world. Before his time glaucoma meant you you had to accept that if you got it you went blind (and some people are born with it) but he refused to accept that and today it is a treatable condition. He said:

Nothing in the world can take the place of persistence; talent will not, genius will not, education will not. Persistence and determination are absolutely omnipotent if you want to get results.

In learning there will be times when you have to struggle, things will seem hard or even impossible but when those times come just remember what Pitts Crick said and you will WIN through.

When you do research you will be using and processing data so it is a good idea to familiarise yourself with the following ideas. It is important that you understand what it is you are dealing with and the concept of measurement requires some scale along which different values can be placed. Three types of scale are possible.

Nominal - a scale used to represent unordered variables. For example we might collect statistics on colour preference. Clearly there is no sense in which a preference for BLUE is greater than RED so in this case any convenient ordering arrangement will do.

Ordinal - a scale used to represent an ordered series of positional relationships. That is where values only indicate position in a series but not absolute values. e.g. examination marks since no one hopes no one would argue that someone who gets 50% knows twice as much as someone who gets 25% or a that a person gaining 100% knows everything.

Interval & Ratio - a scale where a particular interval is the same anywhere on the scale and it is meaningful to refer to zero or say that one value is a certain multiple of another. e.g. distance measurements in meters.

So in processing data it is important you understand these distinctions else for example you might end up processing ordinal data as if it was interval and ratio and inevitably reach the wrong conclusions or if you end up publishing your finding and someone notices what you have done your work and you will be discredited.

Here is a simple self-completion questionnaire which will allow you to asses your 'mindset'. There is nothing sinister here and its intention is to allow those going to College or University to asses how they think and learn. We are all a bit arrogant about our own powers of thinking and ways of learning but its as well to really find out if you really know and can honestly face up to how YOU think and learn as that knowledge may be invaluable to you.

If you wish to try this then send a private email to me by clicking on my board name and include a string of Y/N answers, one for each question shown below. I will then send you a key so that you can interpret your own responses. I am doing it this way so that the questionnaire is not spoiled for later readers - that is, if you know the key before you start then the questionnaire is worthless to you.

Questions - just answer Yes or No as you go though

01. When you do a multi-choice test is your first concern to know if you were right or wrong?
02. If you and others posted answers to a question on a discussion board for tutor comments. Would you mostly only look at just your own entries?
03. It is a positive outlook on life never to attempt anything unless you know you can succeed?
04. The saying “if at first you don’t succeed try, try again” is wishful thinking, it wastes time?
05. There is a saying that “practice makes perfect” so if you get stuck when learning it is best to seek help rather than struggle with it?

06. Success is the key incentives for learning?
07. Is it a sign of weakness to admit in a meeting of colleagues that you made a mistake?
08. Think back to the times things went wrong for you, was your first reaction to make an excuse or blame someone or something?
09. Is success essential to the way you think about yourself, your self esteem?
10. Is it important that others have a high opinion of you?

11. Are you ready to give praise and encourage others?
12. Do you ever say “I was never good at mathematics”, “I cannot spell” or in general there are subject or skills beyond you, you will never be able to do them?
13. Are your choices often conditioned by what others have, say or do?
14. Do you think you are not really very bright and that is just the way it is?
15. Do you have a learning strategy that always works for you?

16. Positive feedback is important in producing better work?
17. Negative feedback is too discouraging to be really helpful?
18. Is it important to you that tutors make it easy to learn?
19. The tutor’s job is to ensure that I learn well?
20. Would you be willing to accept the blame for some mishap, even when it is not entirely your fault so that time is not wasted?

21. Is it true that the more intelligent you are the more successful you are likely to be?
22. I am motivated when I get praised for my work?
23. Is it true that “all work and no play make Jack a dull boy”?

When starting a project it is always good idea to define as best you can a problem your project/dissertation is set to solve or partially solve. It doesn't matter whether your project is about ancient Egyptian invaders, computers or business it is always wise to locate a problem because that gives focus to your work. It is also best to select just one problem of significance and work on that. Students easily overestimate what they can do and if unchecked tend to list dozens of problems that they are going to solve in one project. I once had a student on a Master's course who confidently set out to solve 37 distinct problems but fortunately he came to see how foolish such a list made him look in the eyes of others, not because of the quality of his intellect but because of his inability to see what is possible in the time available.

Presenting Problem
It is difficult to find an adequate and useful definition of the term problem, so it seems best to use a simple definition coined by Professor Peter Checkland at Lancaster University who said a problem is: "a matter of concern or debate amongst situation actors." Actors here means the people who live with the problem and are also the people who would if they could implement a solution.

It follows a problem is an object not an activity. So you as the students must argue from evidence that a problem exists but it is best to end the argument with a short and lucid problem statement such as:

…inventory discrepancies leading to additional costs and delivery delays.
…delays and errors in generating audit request data
…customer complaints about delays in resolving problems via the help desk
…bottlenecks in the repayment system causing customer complaints and miss-payments
…lack of trust in security checking activities
..doubts about the relationship between recorded events and contemporary history in ancient societies
..author credibility in some early accounts of the study of autism

2. WARNING - There is no single place in project inception where more errors are made than in problem definition so do not rush it and do not fall into the trap of thinking it's a trivial matter. So if you have thought of a problem then discuss it with anyone who will listen to see if it makes sense and ask for critical feedback. Post it here to see what others think.

3. Common Errors Reported by Tutors – here is a list of the sort of errors seen in problem defintion:

a. Repeating the setting again - so one might read "my problem is the accounting department". Here the problem is confused with its location.

b. Saying the problem is the topic Area - so one might read "my problem is marketing". Here the problem is confused with a general topic area.

c. Saying the problem is the same as the solution - so one might read "my problem is lack of suitable software". You may be wondering why this is not a problem statement but if you think for a moment you will see it is actually a solution and the reason it is a solution is that one has to ask how the student would have known that the software was not suitable; there must've been some evidence and that evidence would have pointed to a problem which suitable software would have solved.

d. Saying the problem is finding out how to do something so one might write - "my problem is to find out how to reorganise the personell department". Here the confusion occurs because the student is telling us what he is going to do not why he is going to do it; that is the problem itself.

e. Offering an unsuitable problem so one might write "my problem is fuel economy in the A318 airliner" and offered by a student doing medical science or English Literature so that there is no subject area learning involved.

f. Saying the problem is how to make a decision so one might write - "my probem is to decide whether to use process A or process B. It is obvious that we do not knoiw what problem led to the need for such a decision.

g. Saying the problem is that something is missing and this is the same as item c.

Based my last post I thought you might like to see an example of a well-thought out problem definition and in this case its from a Business IT students.

Presenting Problem – The bank is concerned with security for itself and its account holders and is aware of the rise in frauds perpetrated via identity theft or other internet based threats. The Bank, through its account holders and staff have observed that there is an increase in bogus email messages pretending to be from itself to customers and sometimes from customers to staff and we wish to be to deal with this situation and prevent or at least reduce possible fraud routes and events. In this context one particular type of false email message is of serious concern which uses an illegal approach known as “phishing”.

Summarizing the above, I now formulate my problem definition as: Phishing is identity theft using email where a personal message seeks confidential or private information from its recipient whilst posing as a legitimate request. The intention is to fool the recipient of the message into releasing information which can subsequently be used for fraudulent purposes. One might use the simple acrostic FEE-CAP and analyze the phishing problem to arrive at:

Features: it is illegal, intrusive, upsetting, prays on those who trust their fellow man.

Environment: incidents may occur at home or at work and in both these situations one naturally feels comfortable and secure that one’s systems are well protected. However, that context can lull one into trusting the messages one gets and it is exactly that element that the fraudsters want to exploit.

Effects: if phishing is successful the basic effect is a loss of money. However, when that happens there is a ripple effect that destroys confidences in the system, the bank, ones fellow man, the internet and so on.

Cause – the cause is to do with greed or wanting to harm someone and feel, rather sickeningly, pleased in being successful. There is an interesting point here; mostly when you know the cause this helps you to solve the problem but here the root cause is not solvable although its effects are possibly preventable.

Associations: email, chat, on line banking, file sharing and credit cards.

Perspective: in this case I will select a Bank management perspective because the organisation wants to do all it can in terms of prevention.

Referring to my earlier post some have asked for an expansion on the idea of actor and outcome. I advise you to take this to heart as it it crucial to generating a good project that you understand and use these two ideas. Else you may find the old adage is true about you "I aimed at nothing and managed to hit it"

When you do a project or dissertation it is almost always a good idea to focus on the outcome, the object that you will produce (after you have written an introduction, literature Review, research design, displayed your results). For example, at the end of a project you might produce a model or a survey report or a protocol and so on. It follows that if you think this way you should also ask yourself who (the actor) will and can use the outcome you produce.

For example, if you generate the outcome of a survey report on training with a few recommendations then you may find that the training manager (the actor) might be able to use it to bring about useful changes in the training regime. Similarly, if you produce a new model for financial auditing in a small company and you think for a moment you will realise that the model could be used by the auditing team or the auditing manager to set up new processes based on your model. Here are some examples produced by students on a technology course.

Example 1. Problem = students are not able to create a correctly structured Research Question.

Target – that students can present a well structured and lucid Research Question that satisfies the examiners.

Outcome – a set of explanatory notes and model of a Research Question with illustrative examples

Actor - students on the course. If they take the above Outcome (the means to bring about change) and use it in careful study and practice they should be able to achieve the target of a well constructed Research Question (the change we want to bring about) that solves the problem

Example 2. Problem = lack of trust in security checking

Target – security assured network.

Outcome – a revised scope of work and workflow model

Actor - This Outcome (the means to bring about change) will then be used by the outsourced security company team to adjust their testing suite so as to achieve the target of a security assured network (the change we want to bring about).

Example 3. Problem = failures in outsourced software development applications.

Target – applications delivered on time and within budget.

Outcome – a generic model for the management of software application development in outsourcing relationships

Actor - This Outcome (the means to bring about change) will then be used by company based project managers to monitor and control outsourced development projects (the change we want to bring about).

These two terms need to be thought about when doing a project/dissertation because they help you to know how you are actually thinking about the problem you are trying to solve. The vast majority of students can easily recite a definition for induction or deduction but that is often as far as it goes in that they have no idea what such a choice would mean in terms of say the data they would collect.

Be honest and ask yourself - "If my study is inductive, how would that help me define the data". Similarly ask yourself, "if I were deductive, how would that help me define the data". If you do not understand the implications of your choice of induction or deduction on the primary data you define and collect then it probably means you have no clear idea what they mean and that may mean you have a serious weakness in your research plan. I will post some further explanations later but it is best if you look at these example and hopefully you will begin to see how these modes of thinking differ and lead to different data.

Some Examples
1. To be very simple, suppose a crime has been committed somewhere along the A3458 road in West Yorkshire. If he were inductive then the Chief Inspector of police would say to his forensic staff “walk along the road looking for evidence” but if he was deductive he would say to his staff “walk along the road looking for a pair of blood stained overalls and black gloves”. You might now like to think about these two ways of thinking and ask why the Chief Inspector might come to one or other of these ideas.

2. Suppose I want to show that the “Lose Weight Quick” diet works. I can do this by getting volunteers and putting them on the diet and observing if there is any weight loss over some time period. Assuming that there is some weight loss I can use induction (roughly “more of the same”) to infer or predict let us say that usually the diet works. Now you must understand that this is NOT proof; meaning I cannot say on the basis of my observations that the diet will work for everyone, anywhere and for all time. So all I am really able to do here is infer or predict (that its looks likely) that the diet will work in many cases or if you like I have reduced the level of uncertainly about the efficacy of the diet.

You need now to note that no amount of simple weight observations will tell us WHY the diet works. The only way to find out “why” is to have a theory about it and then form a hypothesis and test it. So I might decide that my theory is that obesity is caused by hormonal malfunctioning triggered by the consumption of particular types of carbohydrate containing foods which lead to insulin secretion and that causes fat to be stored. Armed with this theory I can now devise a way to test it and so this now becomes deductive in nature because I am not just inferring a result I am attempting to explain it in a way that I can test.

You might further note that when we used induction it obviously cannot be used to infer or predict that some other diet called “Fight the Flab” will also work. However, once we have a theory that has been substantiated it might then be possible to indeed say that “Fight the Flab” might work because it is based on the same theory.

There are two broad types of research study; which are usually named as follows.

Interventionist – that is when you set up a trial or experiment of some kind that is likely to change the setting and then observe its consequences. As a simple example one could set up a new business process and then see how well it is performing by using for example a questionnaire to gather relevant data.

Observational – here one simply selects a situation and observe it as is stands in some way. For example, if one wanted to find out the general attitude to training in a company one might conduct a series of interviews with both general workers, to see if they felt the training had any value and relevance to their work and managers, to consider if they felt things such as productivity or morale had improved because of the training.

Notice that the type of study has nothing directly to do with how you actually collect data; indeed one can use any data collection method that seems appropriate in the sense that it will help you get accurate and reliable data - in the above examples I used a questionnaire in one case and a seminar in the other. Be imaginative as well as practical, don’t lazily assume in every case that a questionnaire is the only possibility. Here is a list of collection vehicles; but don't assume you know what they mean because the words are familiar - do some reading and find out how they are used and when they are used:

Primary Data Collection Vehicle

Activity logs/skill sheets/Diaries
Document searching
Focus groups
Interviewing
Observation
Portfolios
Questionnaire
Role Playing /Simulation
Seminars
Life Histories
Tests

Sorry posted the same thing twice - perhaps and administrator will delete this one

Thinking Inductively This means making inferences from the observations or experimentation on particular instances (you might loosely say “more of the same”). For example, you notice something that happens and see that it happens often. For example, you notice that people in your company tend to be more productive after IT training so you then make a generalisation using induction and say “training in IT leads to personal productivity increase”. Notice, this is not a law or proof it is a simply an expectation.

The important thing to understand, is that if you think inductively then you are essentially saying that you have no particular view of the problem area and (loosely) you are more or less guessing what data you want and it’s only when you get it that you are able to make inferences from it. Even then we have the problem of how can you logically explain how to make an inference.

It is crucial you are aware that whatever route you take generally nothing has been proved and all you have is an indication. It is not a proof because we cannot know what new tasks or technologies might occur in the future. One of the greatest modern philosophers, Popper, put it like this, “you can never accurately predict the future because it is impossible for men to know now what they, or others men, will know in the future”. It follows that we are never possessed of the data that can allow us to make fool proof predictions about what may lie over the horizon based on our current stock of knowledge.

However, there are problems with induction because it’s impossible to know when you have enough data to be able to draw a conclusion. So if we hypothesis that “all Swans are white” (a common but hypothetical philosophical question) we can go out and look at many Swans but you cannot look at every Swan so how can you be sure? To deal with this there are two well known positions?

Proof by Falsification - Karl Popper pointed out that from a methodological point of view there is little point in using induction but instead accepts the plain truth that we have no certainty. So Popper suggests we change the focus and instead of checking if every Swan is white you ask instead “is this Swan black” which in principle is a much easier question. So if we examine 1000 Swans and find them all white we accept it as likely to be true. So in simple terms looking at falsification is more efficient.

Proof by Inference or Abduction - this is sometimes called “Lipton Inference” after the Cambridge Philosopher Peter Lipton who sadly died in 2007. Essentially he suggested that we form a hypothesis and then define in expectation some data and if we can explain that what we defined is much like the data we actually find we accept it as a kind of proof. Perhaps it can be put more simply as trying to find the best explanation for the evidence we find. It’s as if you are saying, does this defined data fit and explain a situation or phenomenon. As a simple example, if you hypothesise that a new protocol will correct weaknesses in Requirements Gathering and you collected data to get the protocol in a way you can then argue that the protocol explains the variation in the data. One might think of proof by falsification as proof by elimination and proof by inference as proof by explanation.

One must add that it’s is very tempting to think that the more data you have the more right you will become but in fact that often does not seem to be the case and natural randomness begins to play a part. It is also true that when you have a lot of data you tend to start creating theories all the time – that is if there is enough data you are bound to find relationships in it and that is very seductive but often unwise. This is yet another reason to favour deduction if there is any possibility of defining a useful theory.

One final example, suppose we were all sitting together in a classroom and I say to you "I have lost something please help me find it" now if I stop at that point all you can do is look around and when you find something that at least looks reasonable because you know me so you can guess at the sort of thing I might have lost (you would hardly think it was an industrial vacuum cleaner or a set of scaffolding poles) you bring it to me and ask "is this what you lost" - now that is like induction.

Alternatively, you can say something like (a theory), “the silly old codger is always losing his reading glasses” then your task is much easier because now you know what you are looking for, you may not find them or it might not be what I lost but you know what you are looking for and that is like deduction at it can be much much more efficient. There is no right/wrong approach and it’s a matter for you to consider what sort of outlook you are taking.

format_quote Originally Posted by researcher

I'm in postgrad study and finding the 'philosophy' aspect of social sciences difficult to say the least! Is there texts/links you could reccomend inshaAllah? On things like, critical theory, critical realism, ontology, epistemology etc? I'd be very grateful inshaAllah! Although I understand 'some' of the philosophy stuff I'm finding it difficult if not impossible to make the 'link' between my research and the philosophy - HOW do they link? Any tips on a critical lit review??Many thanks

I have taught at all levels but it would be helpful to know what your first degree was if you have one. Philosophy like many subjects can be hard going because it relies on you knowing and understanding several concepts. We all 'do' philosophy everyday but the difference when you study is that you have to learn the language so you can put into words concepts to how you and others think and act.

Everyday we all receive a great quantity of information about the world we live in; generally far more than we need and coupled with this we also have our own experiences. The fact that you experience and appreciate the world does not need to be articulated in your thinking. However our experience provides grounds for knowing and valuing the objects around us and it maybe useful in life and certainly in academic philosophy to conceptualise these things. It is hard to know where to start and the truth is that only by reading widely will you get to be able to use this language meaningfully. So some advice:

1. It is one thing to know what a philosophical term means but you never really understand it until you try to use it. For example, philosophy often speaks about 'value' and you may think you know what that means but the idea of 'value' will only be clear to you when you start using it and identifying values and why something becomes a value. So suppose I say that reading is a value prized by scholars and it is likely you will agree but can you explain it and would that explanation be generalizable to other values? Here we might say that the value represents something good and if it is good then I would like to create or preserve it.

2. Many key ideas in philosophy are to do with a reason and how our experience is connected with both from a theoretical and practical point of view. It is often said that the theoretical reasons are reasons to believe (or you can say justify or explain a belief) and that practical reasons are reasons to act (because my senses give me information). So explaining and justifying are a major concern of rationality.

3. Theoretical reason is it roughly the topic epistemology or the way we go about finding knowledge and this is generally thought to occur in two ways. These are just philosophical ideas and it must not be thought that if one is using empiricism then no logic or thinking processes are involved or if one is being rational you ignore what you see in the world. This is about the driving principle behind the way you might be working.

Empiricism – that is we discover new knowledge by a process of observation and experiment and in technology this is often the most common way to proceed.

Rationalism – the other means of finding knowledge is to use logic and work it out that way. This is often also called a priori reasoning (meaning, roughly, based on what is prior to observational experience). Now this does not mean you ignore the findings of others but it does mean you construct new knowledge just by using your rational powers. Rationalism itself might be theoretical; I form a theory about the distant Universe or practical; I find a reason for taking some action such as buying a new car.

With these two things in mind you can think about induction and deduction but again don't be fooled into thinking that because you can define these you know what they are. Therefore, if you are trying to find out something then to be sure you understand these terms you must be able to say what difference it would make to the way you set about that task.

4. Ontology is loosely the science of what a ‘thing’ is. In this context we are thinking of the ‘thing’ called ‘body of knowledge’ – we try to say what a body of knowledge is (define it if you like). As a simple example, how would you describe the concept 'car' trying to get to its substance and essence. For example for a car we could describe its colour, top speed, engine size, seating capacity and so on. Can we do the same for a given body of knowledge (the one in your Project for example), can we make a list of its features so as to have a description of it, can a knowledge take action, can a knowledge change into something else, does it link to other knowledge, does knowledge have substance and so on.

5. Critical Theory on your course probably designates several generations of German philosophers and social theorists in the Western European Marxist tradition known as the Frankfurt School. According to these theorists, a “critical” theory has a specific practical purpose in transforming society. So so its importance in this philosophical world to have ath eory that explains and hence its use brings about a transformation in society. Think of Marxism or socialism or capitalism or any other ism. From a philosophical point of view theories are deductive so if you like you have decided what the variables are and so you know or expect what the outcome will be. To exaggerate - if your theory says that all the ills of society are caused by people over 6ft tall then the theory says get rid of all tall people and things will be better. So tall people explain (the rationale) societies ills and that leads and inevitability to a desire to get rid of them (actions).

6. So much for the theory but critical realism seems to be about how things work in the real world and which we are able to be discovered but a distinction is made between the world of nature and the world of the social. The first is amenable to a variety of forms of experimental and statistical analyses and the latter is not. It follows the critical realism has its own way of dealing with lets call it data. The reasons offered for this are:

a. The social researcher is part of the world being studied so the act of researching affects what is being researched and its results changes the social world. (there is nothing new here)

b. The world of the social is composed of agents who keep changing things (including their thoughts and values). In short, the social world cannot be as controlled as the objects defined by the natural sciences. (this is a point of view)

So a basic premise of Critical Realists is it the social world cannot be studied with the methods that have proven powerful in analysis, building theoretical explanations and predicting events in the world of natural object. In summary from a research project point of view you need to select one of these critical theories and the use critical realism methods to get the data and make sense of it

7. See my next post for Literature Reviews

8. My knowledge of the Social Sciences is very limited but in terms of starting off with philosophy there are the books I found most useful and influential.

Mark Warnock, An intelligent Person's Guide to Ethics, Duckworth
Karl Popper, The Open Society and Its Enemies, Vol 1 and 2, Routledge
Popper is very clear and his logic in my view faultless but some people hate him and a lot of those people are in Social Sciences - might therefore give you a good angle but Vol 2 likely to be the most valuable to you.

If you can get them now. The first is a very good discussion of the idea of value and the second on Ethics but read Warnock first

E.J. Bond, Reason and Value, CUP
W Lillie, An Introduction to Ethics, University Papers Backs