Reliability and Validity - Developing a Good Instrument.

There are a lot of factors that go into developing a good instrument, like the choice of items and measurement scales, using the appropriate sample for pilot testing, implementing statistical techniques that fit the characteristics of your data, etc. Testing your instrument for reliability and validity can help to assess if your instrument is "good" or "bad" and ultimately help you to know if the interpretation of your data is accurate or misleading.

Testing for reliability and/or validity is not a simple process, and can take years of implementation in different samples to determine. Below, I will list the different types of reliability/validity and how they are assessed.

It's important to remember that reliability is a necessary but NOT sufficient condition of validity. A necessary condition of a statement must be satisfied for the statement to be true and a sufficient condition is one that, if satisfied, assures the statement's truth. In other words you can have an instrument that is reliable but not valid. However, if your instrument is valid, then it HAS to be reliable. It reminds me of the old adage that states: "all poodles are dogs, but not all dogs are poodles". It's the same thing in this case: all valid instruments are reliable, but not all reliable instruments are valid. 


 Definitions:

Reliability - the degree to which an instrument consistently measures whatever it intends to measure. In other words, it's the statistical measure of the reproducibility or stability of the data gathered by your survey.

Validity - the degree to which an instrument measures what it is suppose to measure. If your instrument is valid then you can feel confident in the interpretation of the data. Going back to the adage above, a valid instrument must also be reliable, but what does that mean? Here is another way to interpret it:

Precision (reliability) + Accuracy = Validity

If you spent any time studying the sciences, you were sure to come across precision and accuracy. Although the differences of these terms are clear to me now, they seemed very ambiguous when I was in Chemistry 101, so I'll use the bulls-eye analogy to explain. Precision, AKA reliability, is when all your "hits" are clustered in the same area (the degree to which repeated measures under unchanged conditions show the same result). Accuracy is when all your "hits" are close to the bulls-eye (how close the measurements are to the actual value). With both of these properties together, you have a validity (your "hits" are clustered together around the bulls-eye).
 

Image from: Precise Positioning and Targeting

Types of Reliability:
 
Test-Retest (stability) - the degree to which scores on the same test are consistent over time, in other words, the questions are worded in such a way that cause the respondents to consistently answer the same way. To test this, you would implement the survey to a sample twice and then calculate the correlation coefficients (r) to compare the two sets of responses. Correlation coefficients are considered good if they are 0.70 or above, indicating that the responses are reasonably consistent from one point in time to the other. The trick is determining the amount of time to wait in between implementation. Two weeks is suggested as a good amount because it's long enough for the respondents to forget their answers, but short enough so that they don't gain knowledge or change behaviors before the second survey.

• Intraobserver (intrajudge) - measures the stability of responses from the same person. This is a type of test-retest reliability because it looks at a single individual's score over a period of time. It is also measured using correlation coefficient.

Alternate Form (equivalent-form) - the degree to which two similar forms of a test produce similar scores from a single sample. The two instruments will have the same structure, number of items, reading level, difficulty level, etc; however, each item is not the same. Items differ in wording, but still measure the same idea. You can do this by implementing the survey in two separate samples of the same population, or implement the survey twice in the same sample (as a pre and post-test). Correlation coefficients (r) are compared and high values indicate good alternate-form reliability.

Internal Consistency  - indicates how well different items measure the same issue. It is applied to a group of items that are thought to measure different aspects of the same concept. This is important when measuring the reliability of latent constructs because a single item will not be able to assess concepts such as knowledge, behavior, and attitude. Below are two commonly used methods to assess internal consistency.

• Split-Half - measures internal consistency by comparing two parts of a single instrument. Divide the instrument (or construct items) into two halves, compute each respondent's score on the two halves, correlate both scores. High correlation coefficients indicates high internal consistency.
• Cronbach's Alpha -indicates how well the other items complement each other according to a single construct. This can be used for dichotomous items or longer measurement scales like the Likert scale. High CA values indicate high internal consistency. If your instrument only involves dichotomous responses (e.g. yes or no) then the Kuder-Richardson 20 or KR-20 is another option for this statistic.

Interobserver (interjudge/interrater) - measures how well two or more evaluators agree in their assessment of a variable. It refers to the consistency of two or more independent observers and is usually reported as a correlation coefficient. This type of reliability is used in qualitative studies like interviews, focus groups, or open ended surveys.


Types of Validity:

Face - involves the feedback of untrained reviewers. If you were to categorize validity testing into stages, this would be the first one. You're looking for incorrect spelling and grammar, ambiguous items, confusing layouts, etc. Untrained reviewers will focus on the overall aesthetics of the survey and not the content.


Content - the measure of how appropriate items or scales seem to a set of trained reviewers. Like the term suggests, you're looking at the content of the survey. The more people you can have to look over the survey, the better because each person will point out something different. This should be conducted after you check for face validity and can therefore be referred to as the second stage.

Criterion - the measure of how well one instrument compares to another. This is determined by relating the performance of your instrument to another instrument (the criterion against which the validity of your instrument is judged)

• Concurrent - the comparison of your instrument again another that is considered to be the gold standard for the variable in question. It is calculated using correlation coefficients and high values indicate good concurrent validity.
• Predictive- the degree to which a test can predict how well an individual will do in a future situation. For example, the GRE is suppose to be a good predictor of how well we will do in graduate school. However, I think that many of us will disagree with the GRE's predictive potential, but that's another subject for another time. :) Correlation coefficients are used to compare the initial score with the secondary outcome.

Construct - the degree to which an instrument measures a construct. This is the most important form of validity because it answers the question: is this instrument measuring what is was intended to measure? However, it is also the most difficult form of validity to understand, to measure, and to report.

• Convergent - implies that several different methods for obtaining the same information will provide similar results. Assessing convergent validity is similar to alternate form reliability but is more theoretical. This requires a great amount of work over a long period of time to determine.
• Divergent - measures the ability of an instrument to estimate the underlying truth in a given area. This is also very theoretical and requires a lot of time and work to determine.

References:

Litwin, M.S. (2003). How to assess and interpret survey psychometrics: the survey kit 2. Sage Publications: Thousand Oaks, CA.

Gay, L.R., Mills, G.E., Airasian, P. (2006). Educational research: competencies for analysis and applications. 8th Edition. Pearson Merrill Prentice Hall: Columbus, Ohio.

Definitions

I'm working on compiling definitions of all key terms that I will be using in my research:
environmental education
environmental literacy
experiential education
experiential environmental education

I will probably be changing these definitions ever so slightly as I analyze my data, but here is what I'm using so far along with their references.


Environmental Education
A method for increasing environmental literacy through learner-centered, hands-on experiences that challenge students’ analytical and problem solving skills concerning natural systems and humans’ role in those systems. It incorporates multiple disciplines (math, science, history, economics, political science, English, sociology, etc) along with feelings, attitudes, and perceptions in order to develop responsible citizens and environmental stewards (derived from the North American Association for Environmental Education 2004).


Environmental Literacy
The capacity of an individual to incorporate a broad understanding of how people and societies relate to each other and to natural systems in their daily lives. This requires sufficient awareness, knowledge, skills and attitude, along with the ability to incorporate appropriate environmental considerations into peoples’ lifestyle, career, and community (derived from Elder 2003).


Experiential Education
A philosophy and methodology in which educators purposefully engage with learners in direct experience and focused reflection in order to increase knowledge, develop skills and clarify values (Association of Experiential Education).


Experiential Environmental Education
Environmental education that is purposefully conducted in an outdoor environment (whether natural or man-made) in which students actively participate in a process of exploration and discover of environmental topics, processes, and issues. (This is a working definition that I derived from a qualitative study that I am currently working on. It may change as I analyze further data, but the major constructs are active participation, exploration and discovery, and outdoor environment.)


References:

Association for Experiential Education (AEE). Retrieved March 20, 2010 from: http://www.aee.org/.

Elder, James L. (2003). A field guide to environmental literacy: making strategic investments in environmental education. Environmental Education Coalition and North American Association of Environmental Education: Rock Spring, GA.

North American Association for Environmental Education (NAAEE). (2004). Environmental Education Materials: Guidelines for Excellence. Retrieved March 20, 2010 from: http://naaee.org/npeee/materials_guidelines/guidelines.pdf

Experiential Learning Theory - David Kolb 1984

mark tozer's blog: Experiential Learning

I was planning on doing a short blog explaining Kolb's experiential learning theory however Mark already has a good explanation and there is no reason for me to reinvent the wheel. The model is widely used in outdoor education.

Initially, I was planning on using this model for my dissertation, but recently I found some information that supposedly invalidates Kolb's theory. Instead, I'll probably be using John Dewey's theory because it is more general and because it has the democratic undertones which I believe are important to increasing environmental literacy. That will come soon.

Here is the link to a rebutle of Kolb's theory. Interestingly enough, David Kolb was actually a part of the author's doctoral committee. Makes you wonder how that relationship turned out - ha!

http://www.cc.ysu.edu/~mnwebb/critique.htm

The Adventure Wave - Schoel, Prouty, Radcliffe 1988

The Adventure Wave


Schoel, Prouty, Radcliffe 1988

The Adventure Wave model of experiential learning is comprised of five elements: bedrock, sequencing, briefing, leading, and debriefing. These elements, when thoughtfully put together, create a powerful activity for the learner using a series of peaks and valleys (Schoel, Prouty, & Radcliffe 1988). The five elements are described below.

Schoel et al. (1998) describes bedrock as the foundation of adventure based counseling. The bedrock of a positive and effective learning experience includes specifically defined terms and goals, the development of a proposal and implementation plan, proper leadership and training, adequate resources, initial diagnostic procedures, and proper formation of participant groups. These topics create a solid foundation for developing an experiential curriculum or sequencing play.

After establishing the program’s bedrock, sequencing is used to appropriately order each activity according to the needs of the participants. This is accomplished by, first, using lead-up activities which physically and emotionally prepare the participants for the difficult activity that they will experience. The next part of sequencing is to develop a well thought out plan which properly orders each briefing, activity, and debriefing. Finally, each sequencing plan needs to be flexible due to the nature of adventure based counseling and the fact that each group will have different needs (Schoel et al. 1988).

Once the sequencing play is established, it is important to determine how to brief the participants. Briefing is described as a discussion period in which the group is informed of the rules and the goals of the activity. The topics that are discussed during the briefing directly relate to the upcoming activity (Schoel et al. 1988).

After properly briefing, a group of participants is lead through an activity. During this phase of the Adventure Wave, participants will have a wide spectrum of reactions. They may relax, learn to trust, try new things, and even grow, or, they may rebel against the activity and even the leader. Each activity requires a leader who has an effective leadership style so they can counsel the participants using the appropriate method at the appropriate time (Schoel et al. 1988).

Following the activity, the group moves into a time of reflection known as debriefing. This phase of the wave is typically the hardest for leaders to execute. It involves discussing and dealing with apparent or difficult issues that may have been manifested during the activity. The debriefing session is what makes the activity a meaningful experience for the group (Schoel et al. 1988).




Schoel, J. Prouty, D., & Radcliffe, P. (1988). Islands of healing: A guide to adventure based
counselling. Hamilton, Massachusetts: Project Adventure, Inc.

The Outward Bound Educational Process Model - Walsh and Gollins 1976

The Outward Bound Educational Process Model


Walsh and Gollins 1976

In order for the model to be effective, the tasks need to be introduced incrementally and need to have real consequences. Each problem solving task should be holistic and involve a combination of each individual’s mental, emotional, and physical resources. The conflicts created by the tasks are then reflected on by the student which transfers to the present learning experience to future experiences.



Martin, A., Franc, D., & Zounkova, D. Outdoor and Experiential Learning: An Holistic and Creative Approach to Programme Design. P13

The Active Learning Cycle - Sakofs & Armstrong 1996

The Active Learning Cycle


Sakofs and Armstrong 1996

Although not explicitly stated in the model, Sakof & Armstrong (1996) suggested that the educational process has the following components: the teacher, the experience, immediate applications of knowledge, time for solitude and reflection, adventure, physical fitness, metaphorical significance and teamwork.

"The potency of this general sequence of an experiential lesson in strengthened when the experiences are such that: the task is relevant, thus there must be a meaningful purpose behind the activity, as in real-world applications 2) the task has clear short and long term outcomes known and understood by the learner. 3) multiple routes to meaningful outcomes are available. 4) the learner is empowered to construct emotional and intellectual meanings. 5) leanings build upon learnings."



Sakofs, M., & Armstrong, G.P. (1996). Into the classroom: Outward Bound resources for teachers. Dubuque, Iowa: Kendall/Hunt Publishing Company.

Model for Promoting Learning from Experience - Boud, Cohen, & Walker 1993

Model for Promoting Learning from Experience


Boud, Cohen, Walker 1993

The intended outcome for this model is that the learner will be gain new perspectives from the experience and possibly change their behavior. There are three stages to the model which occur before, during, and after the experience. During the experience it is important to notice, intervene and have a personal foundation of experience. Following the experience the student focuses on feelings and emotions that can enhance further learning.


Boud, D., Cohen, R., & Walker, D. (Eds.). (1993). Using experience for learning. Buckingham: Open
University Press.