Research Methods

[Hugo]

Re: Research Methods: Qualitative Processing

This is perhaps the hardest chapter to write but it is obviously central because here you are exposing for all to see what you really know about research design and the particular research effort undertaking.

Chapter 3 – the Research Design defines a process or model definition to get your intended outcome. This is a critical chapter, so careful work is needed if your design is to be credible. Its KEY element (3.8, 3.9) is the model or process you define to transform your primary data into your intended outcome; if this element is unclear or vague or missing you will not be able to pass the project. Do not fall into the mindless trap of thinking that just be saying “I will look at the primary data and ..” or “I will do a detailed and in-depth analysis of the data...” because such descriptions are so general and so vague that they are meaningless and worthless in explaining what exactly you will do to get to your intended outcome. When writing this chapter it is wise to be clear about two elements:

Results – meaning the primary data as collected has been pre-processed and those results are presented as tables, charts, statistics, and so on.

Outcome – once the primary data has been processed into some usable form (the results) the next step is to generate an outcome. Here is a list of possible forms: An Account of, Best Practice Description, Business Case, Catalogue, Framework, Guidelines, Model, etc​

3.1 Introduction (focused on reminding your readers what your project outcome is supposed to be and a brief outline of what the plan does, similar to what was written in chapter 1)
3.2 Setting Description. This will have been done in chapter 1 but it may be necessary to elaborate on it here
3.3 Discussion of Primary and Secondary data needs
3.4 Research Method consideration of population then selection and Justification. (Survey, Vignette, Case study, experiment etc)
3.5 Discussion of Population and sample frame, independence of sample points, sample precision, sample size, sample selection method (probability: random, systematic, cluster etc, non-probability: convenience, quota, purposive etc, other: event and time sampling), data location, data collection method (observations, interview etc) and data reliability and validity tests.
3.6 Discussion of secondary data collection methods and strategy
3.7 Ethical Review of Outline plan
3.8 Primary Data Pre-processing (define how you will organises and structure your raw data)
To do this well you have to consider carefully what the data is and how best it might be organised using definable processes, statistics, models or other secondary data. For example, you might use tables or summaries or catalogues but always with a mind to help you later on in generating your outcome. It is almost always necessary to cite various authors to explain and lend authority to what you have chosen to do.

3.9 Primary Data Post-Processing (to generate your defined outcome)
In the previous section you would have hopefully worked out how to organise your primary data collection into a whole. Here you must use the organised primary data collection, aided possibly by secondary data, to generate your outcome. It is common to use definable processes, models or secondary data to do this. Again, it is not always necessary to cite various authors to explain and lend authority to what you have chosen to do but it is always a good idea to consider if such academic support is needed.

3.10 Closing remarks and some brief indication using your project plan of how you will manage your: resources, timing and any obvious limitations on your design.

You should be aware that section 3.4 and 3.5 will vary considerably in its detailed structure depending upon the research method chosen. However, selection of a sample frame and the precision with which a sample can be selected are central to having meaningful data and results. However, many project fail badly because 3.5 is inadequate

 

[Hugo]

This is the evaluation chapter and it can be very hard to write but also tends to carry a large proportion of the available marks. The trouble is that it is always difficult to be honestly critical of your own work and instead of evaluating it can end up with you making excuses for things that went wrong or findingt nothing wrong with anything you did.

Chapter 5 - Evaluating your outcome and practice. This is a critical chapter as it shows whether you can look back at your work with a reflective and critical mind. It is very important to realise that evaluating the outcome is a paper exercise because it is done BEFORE the outcome is used whilst the practice evaluation is real because its represents serious reflection on the actual research activity. At this stage the evaluation is project specific; it’s about what you produced and how you produced it and as such it must not stray into conclusions (which is about generalisations of your results).

5.1 Introductory Remarks
5.2 Evaluation of outcome
5.3 Evaluation Practice
5.4 Overview

It is hard to be precise as to what might be covered in each section but the following is a typical way of dealing with the outcome but remember this is all done as a paper exercise by considering in defined ways your outcome and testing it BEFORE it us used. Please be careful here as these factors can look like they just need a yes/no answer but that is not the case you have to test your outcome and argue or make a case for it in each section.

5.2.1 Expected Outcome Functionality and Efficacy (will it do what is intended/needed)
5.2.2 Usability (is the outcome likely to be easy to use by its actors)
5.2.3 Standards (does the outcome fulfil the pertinent requirements of any standards-making bodies)
5.2.4 Expected Effects (does its affect mean changes in policy, process, structure or attitudes)
5.2.5 Systemically Desirable (will the outcome improve the whole problem setting in some way)
5.2.5 Culturally feasible (will the outcome when used prove to be acceptable to those affected by it)
5.2.6 Ethicality (is the use of the outcome ethical, within accepted codes of practice and the law)
5.2.7 Elegant (is the outcome pleasing or more simply did you do a good job in producing the outcome).

Again It is hard to be precise as to what might be covered in each section but the following is a typical way of dealing with project practice but remember this is a real exercise as it was you who used the various techniques and methods to get the outcome. Please be careful here as these factors can look like they just need a yes/no answer but that is not the case you have to test your practice and argue or make a case in each section. The MAIN activity here will focus on the model or method you used to generate your organised and structured data collection from the raw data and then a model or process to generate the outcome from the organised collection and how well they performed and were they really suitable. This is why if you don’t have a clear design for getting your outcome then this section will fail.

There are dozens of areas that might be covered but it’s not usually possible to comment on every single thing you did so it’s best to pick out the outliers or the main things that you leaned or what went wrong but here are several possibilities but generally as a minimum evaluate: your project plan, research design (including data collection and processing) and the literature review.

5.3.2 Literature Preparation (omissions, misunderstandings, sources reliable and current etc).
5.3.3 Primary Data Definition (did you define the data well, did you get good coverage, etc)
5.3.4 Choices Made (reflect on Research Method, Approach and Style, collection, sample size etc).
5.3.5 Collection Protocol (how did this go in practice, were selection criteria accurate, sample size etc)
5.3.6 Pre-Processing (was it easy, were your processing and data organisational idea right or wrong etc)
5.3.7 Outcome Processing (how easy was it to generate your Outcome etc)
5.3.8 Statistical Analysis (if used)
5.3.9 Experience (how did the research experiences/lack of it influence the results/quality of work)
5.3.10 Research Tools and Models asking were the various tools, models, processes useful etc.
5.3.11 Ethics and Anonymity (did your correctly identify all the ethical issues, unforeseen problems etc)
5.3.12 Project Management (how well did planning go, did I contact the right people, timing, mistakes etc).

 

[Hugo]

Chapter 6 - Conclusions and Generalisations. Chapter 5 was about evaluation and that was SPECIFIC to just your project, its data and its outcome. In conclusions you are trying to logically go beyond that and say what it means in the wider world.

The heart of the problem is how we can logically go from specific instances to reach general conclusions. How can we possibly know that what we have observed in our necessarily limited research on given objects and events be enough to enable us to figure out or derive their more general properties. That is, suppose you use your primary data to build a model of human/computer technology relationships. Well that is fine but that model was built using a tiny set from the possible data population so how logically can you get from there to making predication about its use in the wider world of you own company and elsewhere.

Before listing possible sub-sections it is very important that you understand that what you do here to obtain generalizations is almost totally dependent on a thorough literature review coupled with a carefully thought out and executed Research Design; without those you are simply too ignorant of the subject area to use your results and then write anything that might be interesting to read and certainly no way you will add to the general knowledge pool.

6.1 Project Results overview (be careful here not to end up repeating what is in chapter 4)
6.2 Overview of practice and Lessons learned (don’t just repeat the evaluation section here, generalise)
6.3 Further Research
6.4 Generalisations
6.5 Summary

The basic strategy is to use your topic area knowledge and write comments on it by referring to your results. So you might say formulaically “it has been generally agreed that X is a useful strategy to combat Y but based on the results presented here it seems that we also need to include some consideration of Z because....” Now PLEASE be aware this is just a hint as your work may be nothing to do with strategies but the idea or form here is often useful as a way of thinking an item through but be flexible and don’t be afraid to construct logically other ways to express your ideas and certainly notice the ‘because’ as without that you have no argument to present about your findings. Possible areas to do this might be considered in the light of you topic area knowledge and you results:

New meanings, originality, implications, new/modified principles, limitations, new/modified theorisations, indications of best practice, lessons learned, indications of the need for further work, implication for law or standards, warning or cautions, advice, caveats, values, ethics, factors or features including cultural ones, usage and user psychology and other things that might occur to you. So we might write:
​

6.4.1 Implication for X
6.4.2 Indications of Best Practice
6.4.3 Implication for Company Standards on Technology Acquisition
6.4.4 User Psychology Perspective on IT etc

Appendices - This was mentioned earlier but is repeated here. Of these your project specification, project plan and actual data collection are regarded as essential: specification, project plan, Glossary, References list and Bibliography, primary data collection/set. Other items that might be included in an appendix are: Inclusions (copies any relevant documents), Sample Questionnaires, Summary interview transcripts, Details Evaluation scripts, Requirement catalogues, etc

 

[Hugo]

You may have noticed that the very successful Medical Review thread has been closed so I am considering adding Medical Research matters and Statistics here or perhaps consider a parallel thread so like to hear privately or in a post any views or suggestions on this issue

 

[Hugo]

Sample – Constructing a Survey Research Design

This is just an example (given over several postings) of what an acceptable Research Design might look like and is accompanied by some design notes. This example is based on using a survey but even so you may find it useful for other kinds of design in the way it focuses on primary data and how it is processed. The Research Design is the core of what you will be doing in a project - it cannot be looked up in books or on the Internet; you have to work it out for yourselves using the principles and steps that I will give you. We are talking about skills here so this is not about recall as such it’s about practicing and doing it yourself. The steps themselves are easy to remember but generally not easy to do but my advice is to keep each step distinct and try not to run them into one another as that is a certain way to get a complete muddle.

Just to help you in planning you might use the following ideas to decide what kind of thing you intend your research outcome to be but try to focus on just ONE of the following as summing up the main purpose of your study. For example if your intended outcome is a model it means your outcome form is a kind of description. In summary your main project purpose could be about: understanding, exploring, describing, explaining, improving, building or proving something. Many factors will come into play when you are deciding what method to use such as: Context, time, skill, practicalities, Access, cost, quantitative, Qualitative, scale and sensitivity of the data. Once you have explored the above factors make an informed selection of one of the following after careful study of their usage criteria. It is permissible to chose more than one method but it is not recommenced as you have limited time available. The common methods are: Case Studies, Vignettes, Action Research, Experiments, Quasi-Experiments, Surveys, Biographies/History, Grounded Theory, Ethnography and Requirements Gathering

Whenever you ask questions there is always the difficulty of feeling sure that the respondents are answering truthfully and not telling you what they think you want to hear or because they fear what might be done with the data. One must also consider that the question might be poorly worded or the question too difficult and again we might get unsafe results. From an ethical point of view one way of being sure that you can rely on the answers is to preserve anonymity by realizing it can be lost in any of the following 4 ways. Please be aware that if you lose anonymity your results may well be biased.

Lost at the point of collection – for example if I as your tutor send out a questionnaire at the end of a class on Research Methods asking for your opinion of the unit and ask you to send it back to me then the way you fill in the questionnaire might be biased because you know I will know who it came from.

Lost by the method of collection – for example if we collect the data by online means we would give you a password so that a given student cannot submit a questionnaire twice but that means we can or have recorded who you are on the system.

Lost at presentation of results – when the results are presented we have to be careful to remove all identification. For example, suppose I send out a paper questionnaire and on it ask for written comments. It now only makes sense if I send the comments to interested parties and I might very well do that by sending to them copies of the questionnaire. If I have not thought about it I might do that without removing any identification marks or codes.

Lost by classifications – suppose I decide to classify my questionnaire by ethnic origin (or any other thing or things), then I might effectively tell whoever looks at the questionnaires who the respondent was​

It is not possible here to give a deeper treatment of ethics but you will find additional material in the reading list below. In general thinking about the above areas will help you get this right but I do advise you to do further reading as sadly it is common to find that understanding of the nature of ethics is weak.