Question Solved

What metrics should I use to evaluate a text summarization model?

Back to Forum
Starting 40 pts 0 followers
JSS Academy of Higher Education and Research · Posted
Educational Assessment

Context

I am preparing to submit my first research paper to a Q1 journal (IEEE Transactions on Neural Networks). This will be my first submission and I am unsure about a few things.

Questions:

  1. Is it mandatory to submit the code and dataset along with the paper, or is that only required upon acceptance?
  2. My results show a 2.3% improvement over the SOTA baseline. Is that considered significant enough?
  3. How long does the review process typically take for IEEE Transactions journals?

I have read the author guidelines but they are somewhat ambiguous on point 1. Would appreciate guidance from anyone who has published in similar venues.

Sign in to interact

Sign in to join the discussion.

58 Replies

Oldest Top
0
Sowmya Narayanan Active 310 pts · Accepted answer

Flower (flwr) is the most actively maintained and research-friendly federated learning framework right now. My group has been using it for 18 months on a healthcare project with 4 hospital sites.

Strengths:

  • Clean simulation API that lets you test on a single machine before deploying
  • Good support for custom aggregation strategies (FedProx, FedNova)
  • Active Discord community with quick responses from the maintainers

For non-IID data specifically, look at the FedProx strategy.it handles heterogeneous data distributions much better than FedAvg.

0
Chirag Mehta Starting 15 pts · Accepted answer

The short answer: it depends heavily on the journal and field, but here is a general framework.

If your null result is well-powered and the hypothesis was reasonable: Absolutely publish it. Null results in well-designed studies are as valuable as positive results. Journals like PLOS ONE explicitly welcome them.

If your study is underpowered: Be transparent about it. Report effect sizes and confidence intervals, not just p-values. A small, underpowered study with honest reporting is still a contribution.

What kills papers is when limitations are obvious but the authors have not addressed them.

0
Jagdish Rawat · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Amit Joshi Starting 15 pts · Accepted answer

Flower (flwr) is the most actively maintained and research-friendly federated learning framework right now. My group has been using it for 18 months on a healthcare project with 4 hospital sites.

Strengths:

  • Clean simulation API that lets you test on a single machine before deploying
  • Good support for custom aggregation strategies (FedProx, FedNova)
  • Active Discord community with quick responses from the maintainers

For non-IID data specifically, look at the FedProx strategy.it handles heterogeneous data distributions much better than FedAvg.

0
Lalitha Mohan Active 230 pts · Accepted answer

CSIR-UGC NET preparation: past papers are the most important resource. Everything else is secondary.

0
Replying to Lalitha Mohan
Om Prakash Growing 55 pts · Accepted answer

Could you clarify what you mean by results-blind review? I have heard of it but never understood how it works in practice. Wouldn't reviewers need to see results to evaluate whether the methodology is sound?

0
Replying to Lalitha Mohan
Aditya Sharma Starting 45 pts · Accepted answer

This is really helpful context. One more question: for the ethics committee approval, does the institution where the data is collected or the institution where the analysis is performed need to be the primary ethics approver?

0
Sunil Bhattacharya Active 390 pts · Accepted answer

Strongly agree with the point about choosing venues carefully. One paper in NeurIPS > five papers in obscure workshops.

0
Replying to Sunil Bhattacharya
Sunil Bhattacharya Active 390 pts · Accepted answer

This is exactly what I needed. Thank you so much.I have been going back and forth on this for weeks and this clears it up completely.

0
Ishaan Verma Growing 60 pts · Accepted answer

For Zotero vs BibTeX management, I moved to Zotero + Better BibTeX 3 years ago and have not looked back.

My workflow:

  1. Zotero browser connector captures papers instantly from ArXiv, Google Scholar, ACM DL, IEEE Xplore
  2. Better BibTeX generates a clean .bib file that auto-updates whenever I add a paper
  3. Overleaf directly pulls from the .bib file via Dropbox sync

One tip: create a Zotero collection per paper/project. When you export the .bib, export only that collection to keep it clean.

0
Replying to Ishaan Verma
Ravi Patel Starting 25 pts · Accepted answer

This is a known issue with XLM-R on code-mixed data. The MuRIL preprint has a section specifically comparing these.

0
Replying to Ishaan Verma
Lakshmi Devi Growing 75 pts · Accepted answer

Thank you for this detailed answer! A quick follow-up: when you mention cosine learning rate with warm-up, are you warming up the LoRA parameters specifically or the entire model including the frozen backbone?

0
Replying to Ishaan Verma
Jagdish Rawat · Accepted answer

This is exactly what I needed to hear. One clarifying question: you mentioned FedProx handles non-IID data better.does that mean I should switch to FedProx by default, or only if I observe poor convergence with FedAvg first?

0
Quamar Ahmed Growing 65 pts · Accepted answer

+1 to everything said above. My experience was identical.

0
Admin NITK · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Gopal Naidu · Accepted answer

This is exactly what I needed to hear. One clarifying question: you mentioned FedProx handles non-IID data better.does that mean I should switch to FedProx by default, or only if I observe poor convergence with FedAvg first?

0
Revathi Chandrasekaran Active 410 pts · Accepted answer

Strongly agree with the point about choosing venues carefully. One paper in NeurIPS > five papers in obscure workshops.

0
Nandita Ghosh Distinguished 720 pts · Accepted answer

+1 to everything said above. My experience was identical.

0
Disha Malhotra Starting 8 pts · Accepted answer

I have successfully received PMRF (lateral entry, second year). Happy to share what worked in my proposal.

On the research proposal:

  • 60–70% technical approach, 30% context and impact
  • Preliminary results absolutely help. Coursework projects are fine if genuinely relevant
  • The societal impact section should be substantive.PMRF is a national initiative; reviewers want to see why this matters

On citations: Yes, cite your MTech thesis. Framing your PhD as an extension of prior work you did shows continuity of thought.

0
Replying to Disha Malhotra
Chirag Mehta Starting 15 pts · Accepted answer

Much appreciated. I will restructure my proposal along these lines. The point about societal impact being substantive is something I would have gotten wrong.

0
Chirag Mehta Starting 15 pts · Accepted answer

Flower (flwr) is the most actively maintained and research-friendly federated learning framework right now. My group has been using it for 18 months on a healthcare project with 4 hospital sites.

Strengths:

  • Clean simulation API that lets you test on a single machine before deploying
  • Good support for custom aggregation strategies (FedProx, FedNova)
  • Active Discord community with quick responses from the maintainers

For non-IID data specifically, look at the FedProx strategy.it handles heterogeneous data distributions much better than FedAvg.

0
Royal Dsouza Starting 30 pts · Accepted answer

This is exactly what I needed to hear. One clarifying question: you mentioned FedProx handles non-IID data better.does that mean I should switch to FedProx by default, or only if I observe poor convergence with FedAvg first?

0
Suresh Iyer · Accepted answer

For longitudinal data with missing values, mixed-effects models handle this more gracefully than imputation in most cases.

0
Ganesh Menon Growing 140 pts · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Replying to Ganesh Menon
Fatima Sheikh Growing 95 pts · Accepted answer

CSIR-UGC NET preparation: past papers are the most important resource. Everything else is secondary.

0
Replying to Ganesh Menon
Preethi Anand Distinguished 690 pts · Accepted answer

Strongly agree with the point about choosing venues carefully. One paper in NeurIPS > five papers in obscure workshops.

0
Replying to Ganesh Menon
Meghana Rao Growing 170 pts · Accepted answer

Overleaf + GitHub integration is underrated. You get version control and collaboration in one place.

0
Harini Balakrishnan Active 215 pts · Accepted answer

Seconding the recommendation for Zotero. Game changer for managing references across multiple projects.

0
Vikram Bhatia Active 420 pts · Accepted answer

For longitudinal data with missing values, mixed-effects models handle this more gracefully than imputation in most cases.

0
Jagdish Rawat · Accepted answer

Highly recommend checking out the PMRF portal for the actual numbers.they update the stipend structure annually.

0
Divya Krishnan Growing 85 pts · Accepted answer

The Fulbright SoP is very different from a typical PhD SoP. It needs to focus on cultural exchange and diplomacy, not just research.

0
Sowmya Narayanan Active 310 pts · Accepted answer

Highly recommend checking out the PMRF portal for the actual numbers.they update the stipend structure annually.

0
Replying to Sowmya Narayanan
Meena Sharma · Accepted answer

If you are not already using a reading group format with your students, I would strongly recommend it. Forces everyone to articulate what they read.

0
Replying to Sowmya Narayanan
Om Prakash Growing 55 pts · Accepted answer

If you are not already using a reading group format with your students, I would strongly recommend it. Forces everyone to articulate what they read.

0
Replying to Sowmya Narayanan
Sowmya Narayanan Active 310 pts · Accepted answer

Overleaf + GitHub integration is underrated. You get version control and collaboration in one place.

0
Jagdish Rawat · Accepted answer

I have a slightly different take from my experience in industry research. The reproducibility crisis is real but unevenly distributed. Fields with strong engineering culture (computational biology, ML with benchmarks) have actually improved significantly in the last 5 years. The bigger problem is in fields where data sharing is structurally difficult.clinical medicine, behavioral economics.

The ML community's move toward open code and reproducibility checklists has been genuinely effective.

0
Royal Dsouza Starting 30 pts · Accepted answer

The IEEE TPAMI turnaround in my case was 7 months for the first review. Plan accordingly.

0
Priya Nair · Accepted answer

The Fulbright SoP is very different from a typical PhD SoP. It needs to focus on cultural exchange and diplomacy, not just research.

0
Replying to Priya Nair
Pooja Nambiar Growing 115 pts · Accepted answer

If you are not already using a reading group format with your students, I would strongly recommend it. Forces everyone to articulate what they read.

0
Replying to Priya Nair
Jaya Lakshmanan Growing 50 pts · Accepted answer

Marking this as the accepted answer. Incredibly helpful and specific. Really appreciate you taking the time.

0
Pooja Nambiar Growing 115 pts · Accepted answer

Could you clarify what you mean by results-blind review? I have heard of it but never understood how it works in practice. Wouldn't reviewers need to see results to evaluate whether the methodology is sound?

0
Replying to Pooja Nambiar
Ishaan Verma Growing 60 pts · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Replying to Pooja Nambiar
Uday Bose Starting 5 pts · Accepted answer

This is very reassuring. I was worried the null result would kill the paper but your framing around transparency and effect sizes makes a lot of sense.

0
Harini Balakrishnan Active 215 pts · Accepted answer

The ethics review situation in Indian universities is actually more complex than most people realize.

There is no uniform national framework equivalent to the US Common Rule. Each institution has its own Institutional Ethics Committee (IEC) or Institutional Review Board (IRB).

For anonymous survey research with no identifiable data and no vulnerable populations: most Indian IECs would classify this as exempt, but 'exempt' still needs a formal determination from the IEC.it is not self-certifying.

My practical advice: go to your institution's research ethics office now, describe what you did, and request an exempt determination in writing. Better to have documentation than to discover the issue during thesis review.

0
Sunil Bhattacharya Active 390 pts · Accepted answer

Overleaf + GitHub integration is underrated. You get version control and collaboration in one place.

0
Replying to Sunil Bhattacharya
Wajid Khan Growing 110 pts · Accepted answer

If you are not already using a reading group format with your students, I would strongly recommend it. Forces everyone to articulate what they read.

0
Replying to Sunil Bhattacharya
Lalitha Mohan Active 230 pts · Accepted answer

Marking this as the accepted answer. Incredibly helpful and specific. Really appreciate you taking the time.

0
Varsha Pillai Growing 60 pts · Accepted answer

I have a slightly different take from my experience in industry research. The reproducibility crisis is real but unevenly distributed. Fields with strong engineering culture (computational biology, ML with benchmarks) have actually improved significantly in the last 5 years. The bigger problem is in fields where data sharing is structurally difficult.clinical medicine, behavioral economics.

The ML community's move toward open code and reproducibility checklists has been genuinely effective.

0
Hema Suresh Starting 20 pts · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Disha Malhotra Starting 8 pts · Accepted answer

Strongly agree with the point about choosing venues carefully. One paper in NeurIPS > five papers in obscure workshops.

0
Jaya Lakshmanan Growing 50 pts · Accepted answer

I think you are conflating two separate problems here. The publish-or-perish culture creates pressure, yes.but the solution is not to criticize researchers who are navigating an unfair system. The real problem is with evaluation committees who treat publication count as a proxy for research quality. Fix the evaluation, and the incentives change.

Also worth noting: open review has been implemented in some fields and has its own problems. It can disadvantage early-career researchers who fear retaliation from senior colleagues they have reviewed critically.

0
Replying to Jaya Lakshmanan
Rekha Shetty Starting 40 pts · Accepted answer

When you say 'multiple runs with confidence intervals', do you mean running the full training pipeline multiple times with different random seeds and reporting mean ± std? Or something more statistically rigorous?

0
Replying to Jaya Lakshmanan
Anita Rao · Accepted answer

I found that writing a paper abstract first (before the paper) and then reverse-engineering the paper from the abstract helped with clarity enormously.

0
Replying to Jaya Lakshmanan
Hema Suresh Starting 20 pts · Accepted answer

I am in a very similar situation. Would you be willing to share the outline of your PMRF proposal? Not the content.just the section headings and approximate word allocation.

0
Neha Agarwal Growing 90 pts · Accepted answer

I think you are conflating two separate problems here. The publish-or-perish culture creates pressure, yes.but the solution is not to criticize researchers who are navigating an unfair system. The real problem is with evaluation committees who treat publication count as a proxy for research quality. Fix the evaluation, and the incentives change.

Also worth noting: open review has been implemented in some fields and has its own problems. It can disadvantage early-career researchers who fear retaliation from senior colleagues they have reviewed critically.

0
Shiva Prasad Active 480 pts · Accepted answer

This question comes up a lot. The answer really depends on your specific field and what your committee values.

0
Asha Pillai Distinguished 880 pts · Accepted answer

The ethics review situation in Indian universities is actually more complex than most people realize.

There is no uniform national framework equivalent to the US Common Rule. Each institution has its own Institutional Ethics Committee (IEC) or Institutional Review Board (IRB).

For anonymous survey research with no identifiable data and no vulnerable populations: most Indian IECs would classify this as exempt, but 'exempt' still needs a formal determination from the IEC.it is not self-certifying.

My practical advice: go to your institution's research ethics office now, describe what you did, and request an exempt determination in writing. Better to have documentation than to discover the issue during thesis review.

0
Uday Bose Starting 5 pts · Accepted answer

If you are not already using a reading group format with your students, I would strongly recommend it. Forces everyone to articulate what they read.