A practical routine for reading a methods section critically - notes from Yila https://yila.ai https://yila.ai
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-10-22 21:35:24 +02:00
README.md Add research workflow guide 2026-10-22 21:35:24 +02:00

Reading a Methods Section Critically: A Practical Routine

The methods section is where a paper either earns your trust or loses it. The abstract tells you what the authors want you to remember and the discussion tells you what they think it means, but only the methods tell you what was actually done. Many readers skim methods because they are dense and technical, then carry the paper's conclusions into their own work without knowing whether those conclusions are supported. This guide sets out a routine for reading methods critically, the questions to ask for common study designs, and how a source-linked assistant such as Yila can help you keep track of what you found without blurring the line between the paper's claims and your assessment.

Read the methods before the results

A simple habit changes how you read: go from the abstract straight to the methods, and read the results only after you know how the study was designed. If you read the results first, the numbers anchor your judgement and you tend to look for reasons the design supports them. If you read the methods first, you form an expectation of what the study can and cannot show, and then you can check whether the results stay within those limits.

Before you start, write one sentence stating what the paper claims to show. Then, as you read the methods, ask whether the design is capable of showing that. A cross-sectional survey cannot establish that one variable causes another; a small pilot cannot estimate a population effect precisely; a model trained and tested on the same data cannot demonstrate generalisation. Holding the claim in mind keeps the reading focused.

Five questions for any study

Whatever the field, five questions cover most of what matters.

Who or what was studied? Look for the population, how it was sampled and who was excluded. A clinical trial that excluded older patients, a survey distributed through a single online forum, or a benchmark built from one company's data all limit how far the findings travel. Check whether the sample described in the methods matches the sample analysed in the results; attrition and exclusions often appear only in a flow diagram or a supplementary table.

What was measured, and how? Identify the main outcome and how it was operationalised. Was it a validated instrument, a custom questionnaire, an administrative record or an automated metric? Was it measured the same way for every group? Changes in measurement between groups or time points are one of the most common hidden problems.

What was compared with what? Every claim of an effect implies a comparison. Find the comparison group or baseline and ask whether it is fair. A new teaching method compared with no teaching at all will look good; compared with an existing well-run method, it may not.

How were the groups formed? Random assignment, matching, natural variation and self-selection each carry different risks. If participants chose their own group, differences in outcome may reflect who chose rather than what they received. Look for how the authors addressed this, whether by randomisation, adjustment for confounders or sensitivity analyses.

Was the analysis planned? Look for a registration number, a pre-analysis plan or a statement that the primary outcome was specified in advance. If the paper reports many outcomes and highlights the one that reached significance, the finding is weaker than it appears. In computational work, the equivalent question is whether hyperparameters and model choices were fixed before looking at the test set.

Design-specific checks

For randomised trials, check allocation concealment, blinding of participants and assessors, the handling of dropouts and whether analysis was by intention to treat. A trial with large or unequal attrition can give biased estimates even if randomisation was sound.

For observational studies, list the confounders the authors adjusted for and think about the ones they did not. Look at how exposure was measured and whether it was measured before the outcome. Reverse causation is easy to miss when both are recorded at the same time.

For qualitative studies, look for how participants were selected, how data were collected and how themes were developed. Good papers describe the coding process, discuss the researchers' own position and show enough raw material, such as quotations, for you to judge whether the interpretation is grounded.

For machine learning and computational papers, check the data split, whether there is leakage between training and test sets, which baselines were used and whether they were tuned as carefully as the proposed method. Look for variance across runs or seeds; a single number without uncertainty tells you little.

For systematic reviews, check the search strategy, the screening criteria, how risk of bias was assessed and whether heterogeneity between studies was explored. A meta-analysis of poor studies is still a summary of poor studies.

Keep a structured note while you read

Critical reading is easier to sustain if you record your observations in a fixed template. A few fields are enough: the claim, the design, the population, the main outcome, the comparison, key strengths, key limitations and a one-line judgement of how much weight the paper can bear. Write the limitations in your own words and, where possible, note the page or section where you found the relevant detail. That reference is what lets you, or a co-author, check your judgement later.

Over a review, these notes become a small evidence table. They make it easier to see patterns, for example that most studies in an area rely on self-reported outcomes, or that only one trial had adequate blinding. Patterns like these are often more useful to your argument than any single paper.

Separate what the paper says from what you think

The most important discipline is to keep the authors' claims and your assessment in different places. When summarising a paper, it is tempting to write "the intervention improved outcomes" when the paper only showed an association in a non-random sample. Use language that reflects the design: "students who attended tutoring had higher grades" rather than "tutoring raised grades". Your notes should make clear which statements are reported findings and which are your evaluation.

This is also where assistants need care. Tools that summarise papers can be useful for orientation, but a summary that drops the design context can make weak evidence sound strong. Yila is designed to turn papers and data into cited reviews, figures and slides, keeping each statement tied to the passage it came from. That traceability is what makes it compatible with critical reading: when a drafted sentence says a study found an effect, you can open the cited section, check the design and adjust the wording before the claim reaches your own text.

A short routine you can repeat

  1. Read the abstract and write the paper's main claim in one sentence.
  2. Read the methods before the results.
  3. Answer the five questions: who, what measured, compared with what, how grouped, planned or not.
  4. Apply the checks for the specific design.
  5. Read the results and see whether they stay within what the design can show.
  6. Record claim, design, strengths, limitations and weight in a structured note, with page references.
  7. When you write, use wording that matches the design.

Conclusion

Reading methods critically is not about finding reasons to dismiss papers. It is about knowing how much each paper can support, so that the conclusions in your own work are proportionate to the evidence. A short, repeatable routine and a habit of keeping claims and judgements separate make that possible, and a source-linked workflow, whether manual or with an assistant like Yila, keeps the path from your sentence back to the original method visible.