Deliverables

Introduction to the Module

In Module 1, you developed the analytical tools to examine how PTC researchers establish authority through rhetorical and stylistic choices. In Module 2, you learned how literature reviews and citation practices position research within ongoing scholarly conversations, using frameworks like Swales’ CARS model and the CRAAP test to evaluate source quality. In Module 3, you mapped how different epistemologies — from constructivism to positivism — give rise to distinct methodological communities, each with its own standards for what counts as valid evidence and credible knowledge. What you have been developing across these modules is a kind of bilingualism: the ability to step inside a community’s assumptions long enough to evaluate an argument on its own terms, and then step back out. This module asks you to put that bilingualism to work.

Now you will apply all of these competencies by stepping into the role of a peer reviewer. You will assess whether researchers’ methods align with their questions, whether their evidence warrants their conclusions, and whether they acknowledge the epistemological assumptions and limitations of their approach. Peer review is what all skilled investigators do when they evaluate knowledge claims, whether they are researchers, analysts, consultants, journalists, or professionals making evidence-based decisions.

Peer Review and the Construction of Authority

Research gains authority when it has been vetted by peers who share the epistemological assumptions and methodological standards of a given community. Peer review is the process through which methodological communities assess whether investigators have followed established methods, interpreted evidence appropriately, and acknowledged the limitations of their approach. A study’s credibility depends not on the credentials of its authors alone but on whether the research community recognizes the work as meeting its standards for valid inquiry. As the ACRL Framework for Information Literacy reminds us, authority is constructed and contextual — what counts as authoritative evidence in one community may not transfer seamlessly to another (see Credibility & Authority).

This means that when you critique a study, you are asking: Has this researcher followed the methods that their methodological community considers valid? Does their evidence meet the standards their audience expects? Have they situated their work within the scholarly conversation in ways that demonstrate awareness of what has come before? Authority is the goal — and peer review is the mechanism through which authority is established, challenged, and refined.

The story about the gunslinger and the clown captures the gist of methodological critique: disagreement often isn’t about what happened but about how we decide what counts as a valid truth claim. What looks like victory to one community can seem irrelevant — or absurd — to another. Consider a research question such as, “How is AI transforming human cognition, agency, and work — and how do we know?” This is a wicked question — complex, multidimensional, resistant to simple answers. It spans the cognitive effects of offloading thinking to machines, the implications for human autonomy and decision-making, and the reshaping of labor markets and professional work. Think tanks and university research centers are actively investigating these questions, including the Stanford Digital Economy Lab and Elon University’s Imagining the Digital Future Center. Different methodological communities approach this question with different methods, assumptions, and standards of evidence. (For a detailed overview, see Methodological Communities.)

A classic study by Fiske and Fogg (1990) analyzed 402 peer reviews of psychology journal submissions and found that two reviewers of the same paper typically had no critical points in common. Reviewers didn’t overtly disagree; they simply noticed different things. This finding underscores an important lesson: critique is perspectival. Your task is not to produce the “correct” critique but to offer a rigorous, well-reasoned evaluation from your perspective as an informed reader.

Significance — Why This Module Matters

In an age when facts compete with opinions and algorithms amplify belief over evidence, being literate means more than locating sources — it means understanding how knowledge is constructed, contested, and performed through language. Every research study is a rhetorical act, shaped by the questions it asks, the evidence it values, and the worldview it assumes. What counts as truth in one community may look like bias, speculation, or irrelevance in another.

Being able to assess the authority of a research study is therefore a critical literacy — especially in an era of information overload and predatory journals that imitate peer-reviewed venues while promoting ideology or profit. Even reputable studies can suffer from methodological weaknesses such as misaligned questions and methods, flawed data collection, overgeneralization, or ethical lapses. Such problems can produce misleading conclusions that ripple across public policy, industry, and education.

Required Readings

• Methodological Communities
• Credibility & Authority
• Critique – A Research-based Guide to Criticism
• Methodological Pitfalls: Common Flaws Across Research Communities
• Methodological Pitfalls: Flaws Unique to Specific Research Communities
• Fiske, D. W., & Fogg, L. (1990). But the reviewers are making different criticisms of my paper! American Psychologist, 45(5), 591–598. [PDF provided in course materials]
• Facts vs. Opinion• News vs. Opinion

Guidelines for the Creative Challenge

Study selection is required, not optional. You must select two studies from the list below. Do not substitute outside sources — the assigned studies were chosen because they represent distinct methodological communities and allow for meaningful comparison. Submitting a critique of a study not on this list will result in a score of zero for that critique.

You are encouraged to select studies from different methodological communities to experience the contrasts firsthand.

The Scientists (Quantitative Empiricists)

• Kosmyna, N., et al. (2025). Longitudinal study on AI-assisted writing: Cognitive adaptation and agency in human–AI co-creation. arXiv. https://arxiv.org/abs/2506.08872
• Brynjolfsson, E., Chandar, B., & Chen, R. (2025). Canaries in the coal mine? Six facts about the recent employment effects of artificial intelligence. Stanford Digital Economy Lab. https://digitaleconomy.stanford.edu/research-area/economics-of-transformative-ai/

The Interpreters (Qualitative Empiricists)

• Ranganathan, A., & Ye, X. M. (2026, February 9). AI doesn’t reduce work—it intensifies it. Harvard Business Review. https://hbr.org/2026/02/ai-doesnt-reduce-work-it-intensifies-it

The Pragmatists (Mixed Methods Researchers)

• Massenkoff, M., & McCrory, P. (2026, March 5). Labor market impacts of AI: A new measure and early evidence. Anthropic. https://www.anthropic.com/research/labor-market-impacts
• Anthropic. (2026, January). Anthropic Economic Index: Insights from Claude usage. Anthropic. https://www.anthropic.com/research/anthropic-economic-index-january-2026-report
• Tomlinson, K., et al. (2025). Working with AI: Measuring the applicability of generative AI to occupations. arXiv. https://arxiv.org/abs/2507.07935
• Anderson, J., Rainie, L., & Vogels, E. A. (2023). The future of human agency. Pew Research Center & Elon University. https://www.elon.edu/u/imagining/surveys/xv2023/the-future-of-human-agency-2035/

The Scholars (Textual/Synthesis Researchers)

• David, L., Vassena, E., & Bijleveld, E. (2024). The unpleasantness of thinking: A meta-analytic review of the association between mental effort and negative affect. Psychological Bulletin. https://www.apa.org/pubs/journals/releases/bul-bul0000443.pdf

A Note on Communities, Epistemologies, and What Gets Confused

Before you select and critique your studies, take a moment to review two distinctions that trip up many readers.

Epistemological assumptions vs. study findings. A study’s epistemological assumptions are the underlying beliefs about how knowledge is constructed — beliefs the researcher brings before data collection begins. The findings are what the data revealed. These are not the same thing. When a quantitative study reports that AI adoption reduced employment in a given sector by 12%, that percentage is a finding. The assumption that human behavior can be measured objectively, that patterns generalize across populations, and that correlation can support causal inference — those are epistemological commitments. Confusing the two leads to critiques that summarize conclusions rather than interrogate methods.

The Scholars vs. the Synthesizers. These two communities are easy to conflate, especially when a study draws on multiple existing sources. Scholars work within hermeneutic and critical theory traditions — their primary methods are textual analysis, archival research, and philosophical argument. They interpret meaning from texts. Synthesizers (sometimes called Pragmatists in mixed-methods traditions) integrate multiple data types and epistemological frameworks to address complex, real-world problems. A meta-analysis that statistically aggregates results from dozens of quantitative studies belongs to the Synthesizer community — not the Scholars — because its primary method is quantitative integration, not interpretive textual analysis. If you’re unsure which community a study belongs to, ask: What kind of evidence does the study treat as authoritative, and what does it do with that evidence?

The Pragmatists and “practical research.” The Pragmatist community is defined by its epistemological commitment to integrating multiple perspectives to address problems that resist single-method solutions — not simply by addressing real-world problems. Applied research and basic research are categories that describe a study’s purpose (practical application vs. knowledge generation); they say nothing about the epistemological tradition. A quantitative study of labor market outcomes is applied research. It is not, by virtue of being practical, a Pragmatist study. The Pragmatists are identified by their use of mixed methods and their deliberate combination of epistemological frameworks. See Applied Research and Basic Research for a fuller explanation of that distinction.

Guidelines for Each Critique

Write each critique as if you were a peer reviewer offering constructive feedback to the study’s authors. Your tone should be professional, balanced, and specific—acknowledging strengths while identifying areas for improvement.

Summary of Methods (~50–100 words)

Briefly describe the study’s approach to data gathering, analysis, and reporting. Identify which methodological community the study belongs to and note the epistemological assumptions that inform its approach. This summary demonstrates your understanding of the methods before you critique them.

A note on citation practice: When you introduce a source in your critique, lead with the author’s last name and year in parentheses — not the article title. In APA 7 style, this looks like: Kosmyna et al. (2025) examined… or As Ranganathan and Ye (2026) argue… Leading with the title rather than the author signals unfamiliarity with scholarly citation conventions and will affect your score on Clarity & Professional Communication

Methodological Critique (~400 words)

Critique the study’s methods, addressing both strengths and weaknesses. Use Methodological Pitfalls: Common Flaws Across Research Communities and Methodological Pitfalls: Flaws Unique to Specific Research Communities to guide your evaluation.

You may consider the following questions, but you are not expected to address all of them. Focus on the issues most relevant to your chosen study:

Are the methods transparent? Could another researcher understand how data was gathered, analyzed, and interpreted? (Note: Interpretivist researchers may prioritize reflexivity and thick description over strict replicability.)

• Are the methods transparent? Could another researcher understand how data was gathered, analyzed, and interpreted? (Note: Interpretivist researchers may prioritize reflexivity and thick description over strict replicability.)
• Do the methods align with the research question? Does the study’s design actually address what it claims to investigate, or is there a mismatch between purpose and method? Do the conclusions follow from the evidence? Are there unwarranted causal claims, or do the authors confuse correlation with causation? Does the study avoid overgeneralization? Are findings appropriately bounded, or do the authors extrapolate from limited samples to broad populations?
• Does the study engage with counterarguments? Do the authors consider alternative interpretations and contradictory evidence, or do they cherry-pick sources that support their claims?
• Are limitations acknowledged? Do the authors recognize constraints on their findings, including potential biases in data gathering or interpretation?
• Is the sample adequate for the claims being made? Is the sample too small, too restricted, or not representative of the population to which conclusions are applied? Are findings adequately interpreted? Do the authors explain what their results mean, or are findings left unclear or unexplored?
• Does the study respect the epistemological assumptions of its methods? For example, does a qualitative study claim generalizability that requires statistical warrant? Does a quantitative study make interpretive claims about meaning without appropriate evidence?
• Has the researcher established authority? Have the authors followed the established methods and discourse conventions that their methodological community considers valid? Have they situated their work within the scholarly conversation in ways that demonstrate awareness of prior research? (See Credibility & Authority.)

Submission Instructions

1. Format: Your final critiques must be saved as a .pdf and uploaded directly to Canvas.
2. Header: Place your Name and Word Count (minimum 1,000 words for both critiques combined, excluding references) in the top-left corner of the first page.
3. Upload your submissions to Canvas in .pdf by the due date listed in the Canvas calendar.

Evaluation Criteria

Summary of Methods

Accurately identifies the study’s methodological community; clearly describes how data was gathered, analyzed, and reported; demonstrates understanding of the methods sufficient to critique them.

Quality of Critique

Identifies relevant methodological pitfalls—such as overgeneralization, confusing correlation with causation, cherry-picking evidence, inadequate sample size, lack of transparency, unacknowledged limitations, or mismatch between methods and research question; engages substantively with both strengths and weaknesses; considers alternative interpretations; assesses whether the researchers have established authority by situating their work within the scholarly conversation; maintains a constructive, peer-review tone throughout.

Clarity & Professional Communication

Writing is clear, concise, and well-organized; follows formatting guidelines (name, word count, PDF format, APA 7 citations); logical flow; error-free prose; authentic voice evident.

Guidelines for the Metacognitive Footnote

For detailed guidance, examples, and the complete list of legitimate AI roles, see Metacognitive Report – AI Writing Ethics: Balancing Agency, Voice & Disclosure.

Your report must include:

1. Header — Beneath your title, record left-justified:

• Word Count / Name / GenAI Tools Used / Chat Log Links

2. GenAI Usage Table(s) — One table per tool with these columns:

• Step in Writing Process (Prewriting, Drafting, Revising, etc.)
• Number of Chats
• Primary Purpose(s) (Thought Partner, Research Assistant, Teaching Assistant, etc.)
• Notes on Use (2–3 sentences: what you asked, what the AI gave you, and whether you accepted, revised, or rejected it — and why)

3. Critical Reflection (minimum 250 words) — Explain:

• Which roles AI played and why
• At least one moment where you rejected or corrected AI output
• How AI helped you learn something you then applied independently
• Where you made decisions AI could not make for you

Submission Guidelines

Upload your report along with your assignment to Canvas by the required due date.

Related Exercises

• Methodological Pitfalls: Flaws Unique to Specific Research Communities
• Guidelines for Summary and Synthesis of Readings — Methodological Critique