6-Evaluating Sources
Evaluating for Relevancy
Relevant sources are those that pertain to your research question. You’ll be able to figure that out fairly quickly by reading or skimming particular parts of sources and maybe jotting down little tables that help you keep track. We’ll show you how below, including where to look in specific kinds of sources and what questions to ask yourself as you do.
One thing to consider early on as you make inferences about relevancy is the effect that timeliness, or a source’s currency, should have on deciding whether a source is relevant. Your research question will determine that.
For instance, if your research question is about the life sciences, you probably should consider only the most recent sources relevant because the life sciences are changing so quickly. There is a good chance that anything but the most recent sources may be out of date. So aim for sources no more than 5 years old. (An example discipline that calls for even newer sources is computer security.)
But suppose your research question is about the Edo Period in Japan (1603-1868) or about Robert Falcon Scott, who explored the Antarctic from 1901-1913. In these cases, an item from 1918 might be just as useful as an item from 2018 (although new information may have been found in the 100 year gap). But something from 1899 about Antarctica or from 1597 about Japan would NOT be current enough for these research questions.
These example research questions also give you two more clues about how to treat the timeliness or currency of sources as you consider relevance:
- Because of how long ago they lived or occurred, it would be unusual for many sources on Robert Scott or the Edo Period to have been published very recently. So, unlike sources for the life sciences, whether a source is very recent should probably not determine its relevancy to those research questions.
- Primary sources might be considered especially relevant to all three research questions. Life science journal articles that provide research findings for the first time count as primary sources. And primary sources (such as Scott’s diaries and expedition photographs, as well as paintings, literature, clothing, and household items from the Edo Period) go a long way to explain faraway people and times. (See Primary, Secondary, & Tertiary Sources.)
EXAMPLE: Currency
Check out how currency is handled on TED. This site provides videos of speakers talking about new ideas in technology, entertainment, and design. (That’s what TED stands for.) Some videos are labeled “Newest Talks,” and TED tells when every video was recorded.
For your sources for which timeliness matters, see the section Where to Look, which includes where to look in websites, articles, and books for information about a source’s currency.
Time-Saving Tips
Instead of thinking you have to read all of every source in order to figure out whether it is relevant, read or skim only parts of each source. If you’re looking at the right parts, that should give you enough information to make an educated guess about relevancy.
But what should you be looking for as you do that reading and skimming? One way to figure that out is to first parse your research question so that you can figure out its main concepts. (This is like identifying main concepts in your research question in order to search precisely.)
For instance, suppose your research question is: How does having diverse members in a group increase the critical thinking of the group?
What are this question’s main concepts? Our answer is: group diversity and critical thinking.
So when trying to judge which sources are relevant to these main concepts, you would assess whether each source you have found pertains to at least one of these concepts. We recommend you jot down a little table like the one in the example below to keep track of which sources address each main concept.
To be considered relevant to your research question, a source wouldn’t necessarily have to cover all of your main concepts, but finding sources that do is the ideal. Otherwise, you just have to make do with what you have found. Don’t forget that each source would have to pass the currency test, too, if currency is important to your research question. So it’s wise to record your decisions about the sources’ currency on your tables, too.
EXAMPLE: Sources’ Main Concepts and Currency
Research question: How does having diverse members in a group increase the critical thinking of the group?
| Currency Okay | Group Diversity | Critical Thinking | |
| Source A title | X | X | |
| Source B title | X | ||
| Source C title | X | X | X |
The table in this hypothetical example indicates that both Sources A and C are relevant because each pertains to at least one main concept from the research question. Currency doesn’t seem to matter much to our research question, so all three sources were marked current. But since currency is all that Source B has to offer, it is not relevant for this project.
If you do make little tables for relevance, it’s probably a good idea to hang on to them. You might find them helpful later in your research process.
Where to Look in Websites, Articles, and Books
The information below tells where to look and what questions to ask yourself to assess three kinds of sources’ relevancy to your research question. Whatever you do, don’t evaluate a source just by looking at a website’s name or the title of another source. Look beyond the title and use critical thinking.
Save time by looking in particular places in sources for information that will help you figure out whether the source is relevant to your research project. Much of our advice below comes from “Speedy Reading” in The Craft of Research, second edition, by Wayne Booth, Gregory Colomb, and Joseph Williams, 2003, pp. 108-109.
On a website, check the name of the website and its articles for clues that they contain material relevant to your research question. Consider whether time should have an impact on what information can be considered relevant. If so, skim any dates, datelines, What’s New pages, and press releases to see whether any website content works with the time considerations you need. Page creation or revision dates that you find can also help.
Skim any site map and index on the website for key words related to your research question. Try the key words of your research question in the search box. Do you see enough content about your keywords to make you think parts of the website could be helpful?
For an article, think about the title. Does it have anything to do with your research question? Consider whether time should have an impact on what sources can be considered relevant. If so, is the publication date within your parameters? Also skim the abstract to see whether the article works with the time considerations you need. For instance, if there is a time period in your research question, does the article address the same time period or was it created in that time period?
Look at the abstract and section headings in the article to locate the problem or question that the article addresses, its solution, and the outline of the article’s argument for its main claim. Can those help answer your research question? Do they make it seem the article will give you information about what others have written about your research question? Do they offer a description of the situation surrounding your research question?
Do the article’s introduction and conclusion sections help you answer your research question and/or offer a description of the situation surrounding your question so you can explain in your final product why the question is important? Check whether the bibliography contains keywords related to your research question. Do the sources cited by the bibliography pertain to your research question?
For a book, check whether the title indicates the book could be about your research question. Consider whether time should have an impact on what sources can be considered relevant. If so, is the publication date or copyright date (usually listed in the library catalog or on the back of the book’s title page) too early or late for any time constraints in your research question? Maybe it is just right. Also skim some of the preface and introduction to see whether the book works with the time considerations you need.
For help answering your research question, skim the book’s table of contents and any summary chapters to locate the problem or question that the book addresses, its solution, and the broad outline of the book’s argument for its main claim. Do they also give you information about what others have written about your research question? Do they offer a description of the situation surrounding your research question? Look for your key words in the bibliography. Do the sources cited pertain to your research question? Skim the index for topics with the most page references. Do the topics with the most page references pertain to your research question?
ACTIVITY: Follow a Title’s Clues for Relevance
Open activity in a Web browser.
In the previous activity, you evaluated for currency and relevance the titles of three sources for the research question: How does “prospect theory” in behavioral economics help explain medical doctors’ decisions to favor surgery or radiation to cure cancer in patients? The correct answer was [C] “Cancer Treatment Prescription–Advancing Prospect Theory beyond Economics,” in Journal of The American Medical Association Oncology, June, 2016. In the next activity, you will practice evaluating for relevance beyond the title using this source.
ACTIVITY: Connecting the Dots beyond the Title
Open activity in a Web browser.
Instructions: Now you can practice evaluating for relevance beyond the title for this research question: How does “prospect theory” in behavioral economics help explain medical doctors’ decisions to favor surgery or radiation to cure cancer in patients?
Judging by the title, the most relevant source for that research question seemed to be a journal article called “Cancer Treatment Prescription–Advancing Prospect Theory beyond Economics,” in Journal of The American Medical Association Oncology, June, 2016. (Note to students: This article and abstract are fictitious.)
Read the abstract of the article below. Then decide whether this source is relevant to your research questions above. That is, might the article help you meet any of your project’s information needs about the research questions? If there is at least one need it can help meet, then you should judge the article relevant.
Compare your answer to the activity with our feedback at the bottom of the page.
As usual, your information needs are:
-
To learn more background information.
-
To answer your research question.
-
To convince your audience that your answer is correct or, at least, the most reasonable answer.
-
To describe the situation surrounding your research question for your audience and explain why it’s important.
-
To report what others have said about question, including any different answers to your research question.
Abstract for “Cancer Treatment Prescription–Advancing Prospect Theory beyond Economics”
Importance Cancer Treatment is complex. We expect oncologists to make treatment decisions according to definitive standards of care. Finding out that prospect theory demonstrates that they react very much like most other people when deciding to recommend surgery or chemotherapy for their patients indicates that more self-reflection on oncologists’ part could help patients make better decisions. (Prospect theory describes how people choose between alternatives that have risk when the probability of different outcomes is unknown.)
Objective To show whether prospect theory applies to how oncologists framed their recommendations for surgery or chemotherapy for patients in good condition and bad condition.
Design, Settings, and Participants Records of 100 U.S. oncologists were examined for the years 2014 and 2015, which documented patient conditions and the way oncologists framed their recommendations regarding surgery or chemotherapy. Thus, a quasi-experimental ex post facto design was used for the study.
Main Outcomes and Measures This study explored the relationship between the way in which the oncologists “framed” the choice of surgery or chemotherapy as they made recommendations to patients, to patients’ conditions, and the choice actually made. Those results were compared to what prospect theory would predict for this situation.
Results Physicians seemed to present their recommendation of surgery or chemotherapy in a loss frame (e.g., “This is likely to happen to you if you don’t have this procedure”) when patients’ conditions were poor and in a gain frame (e.g., “By having this procedure, you can probably dramatically cut your chances of reoccurrence”) when their conditions were less poor. These results are what prospect theory would have predicted.
Conclusions and Relevance This study opens up the possibility that, as described by prospect theory, a person’s choice of framing behavior is not limited to how we naturally act for ourselves but includes how we act for other people, as the oncologists were acting on behalf of their patients. More research is necessary to confirm this line of evidence and determine whether oncologists’ decision making and framing is the most effective and entirely according to the best standards of care.
Feedback Was this article relevant to the research question? Answer: YES, this source could meet multiple information needs. According to the abstract, the objective focuses on oncologists and how they frame their recommendations for surgery or chemotherapy, which are the main concepts of the research question. The results of the study provide evidence that answers the research question and that should convince your audience that doctors use prospect theory in their decision-making, whether they realize it or not. Also, the “conclusions and relevance” of the study reports what others have said about the implications of the results, and explains why it’s important to conduct more research on how doctors frame their decisions.
