Question Solved Locked

How to measure the statistical significance of A/B test results in ML experiments?

Back to Forum
Starting 30 pts 0 followers
Indian Institute of Technology Bombay · Posted
IoT & Embedded Systems Higher Education Policy Natural Language Processing

Context

I am preparing to submit my first research paper to a Q1 journal (IEEE Transactions on Neural Networks). This will be my first submission and I am unsure about a few things.

Questions:

  1. Is it mandatory to submit the code and dataset along with the paper, or is that only required upon acceptance?
  2. My results show a 2.3% improvement over the SOTA baseline. Is that considered significant enough?
  3. How long does the review process typically take for IEEE Transactions journals?

I have read the author guidelines but they are somewhat ambiguous on point 1. Would appreciate guidance from anyone who has published in similar venues.

Sign in to interact

Sign in to join the discussion.

43 Replies

Oldest Top
0
Pooja Nambiar Growing 115 pts · Accepted answer

For Zotero vs BibTeX management, I moved to Zotero + Better BibTeX 3 years ago and have not looked back.

My workflow:

  1. Zotero browser connector captures papers instantly from ArXiv, Google Scholar, ACM DL, IEEE Xplore
  2. Better BibTeX generates a clean .bib file that auto-updates whenever I add a paper
  3. Overleaf directly pulls from the .bib file via Dropbox sync

One tip: create a Zotero collection per paper/project. When you export the .bib, export only that collection to keep it clean.

0
Xena D'Souza Starting 25 pts · Accepted answer

Strongly agree with the point about choosing venues carefully. One paper in NeurIPS > five papers in obscure workshops.

0
Suresh Iyer · Accepted answer

This is exactly what I needed to hear. One clarifying question: you mentioned FedProx handles non-IID data better.does that mean I should switch to FedProx by default, or only if I observe poor convergence with FedAvg first?

0
Bhavana Reddy Starting 30 pts · Accepted answer

This question comes up a lot. The answer really depends on your specific field and what your committee values.

0
Neha Agarwal Growing 90 pts · Accepted answer

The ethics review situation in Indian universities is actually more complex than most people realize.

There is no uniform national framework equivalent to the US Common Rule. Each institution has its own Institutional Ethics Committee (IEC) or Institutional Review Board (IRB).

For anonymous survey research with no identifiable data and no vulnerable populations: most Indian IECs would classify this as exempt, but 'exempt' still needs a formal determination from the IEC.it is not self-certifying.

My practical advice: go to your institution's research ethics office now, describe what you did, and request an exempt determination in writing. Better to have documentation than to discover the issue during thesis review.

0
Pooja Nambiar Growing 115 pts · Accepted answer

The ethics review situation in Indian universities is actually more complex than most people realize.

There is no uniform national framework equivalent to the US Common Rule. Each institution has its own Institutional Ethics Committee (IEC) or Institutional Review Board (IRB).

For anonymous survey research with no identifiable data and no vulnerable populations: most Indian IECs would classify this as exempt, but 'exempt' still needs a formal determination from the IEC.it is not self-certifying.

My practical advice: go to your institution's research ethics office now, describe what you did, and request an exempt determination in writing. Better to have documentation than to discover the issue during thesis review.

0
Neha Agarwal Growing 90 pts · Accepted answer

The Fulbright SoP is very different from a typical PhD SoP. It needs to focus on cultural exchange and diplomacy, not just research.

0
Meghana Rao Growing 170 pts · Accepted answer

The ethics review situation in Indian universities is actually more complex than most people realize.

There is no uniform national framework equivalent to the US Common Rule. Each institution has its own Institutional Ethics Committee (IEC) or Institutional Review Board (IRB).

For anonymous survey research with no identifiable data and no vulnerable populations: most Indian IECs would classify this as exempt, but 'exempt' still needs a formal determination from the IEC.it is not self-certifying.

My practical advice: go to your institution's research ethics office now, describe what you did, and request an exempt determination in writing. Better to have documentation than to discover the issue during thesis review.

0
Admin NITK · Accepted answer

The IEEE TPAMI turnaround in my case was 7 months for the first review. Plan accordingly.

0
Replying to Admin NITK
Indira Balan · Accepted answer

Could you clarify what you mean by results-blind review? I have heard of it but never understood how it works in practice. Wouldn't reviewers need to see results to evaluate whether the methodology is sound?

0
Replying to Admin NITK
Nandita Ghosh Distinguished 720 pts · Accepted answer

The Fulbright SoP is very different from a typical PhD SoP. It needs to focus on cultural exchange and diplomacy, not just research.

0
Replying to Admin NITK
Hemant Patwa · Accepted answer

This is very reassuring. I was worried the null result would kill the paper but your framing around transparency and effect sizes makes a lot of sense.

0
Sanjay Kumar Growing 145 pts · Accepted answer

CSIR-UGC NET preparation: past papers are the most important resource. Everything else is secondary.

0
Replying to Sanjay Kumar
Harini Balakrishnan Active 215 pts · Accepted answer

Thank you! The reference to MuRIL is particularly useful.I had not considered it as an alternative to XLM-R for Indic languages.

0
Replying to Sanjay Kumar
Shiva Prasad Active 480 pts · Accepted answer

This is exactly what I needed to hear. One clarifying question: you mentioned FedProx handles non-IID data better.does that mean I should switch to FedProx by default, or only if I observe poor convergence with FedAvg first?

0
Replying to Sanjay Kumar
Uday Bose Starting 5 pts · Accepted answer

This question comes up a lot. The answer really depends on your specific field and what your committee values.

0
Ishaan Verma Growing 60 pts · Accepted answer

Respectfully, I think this framing misses the systemic dimension. Individual researchers cannot solve a collective action problem. If you submit to open-access journals and your colleague submits to Nature, your colleague gets the promotion. Until evaluation criteria change at the institutional level.which requires policy intervention.individual choices have minimal impact.

0
Rajesh Kumar · Accepted answer

I think you are conflating two separate problems here. The publish-or-perish culture creates pressure, yes.but the solution is not to criticize researchers who are navigating an unfair system. The real problem is with evaluation committees who treat publication count as a proxy for research quality. Fix the evaluation, and the incentives change.

Also worth noting: open review has been implemented in some fields and has its own problems. It can disadvantage early-career researchers who fear retaliation from senior colleagues they have reviewed critically.

0
Rajesh Kumar · Accepted answer

Open access is the right direction but the APCs are prohibitively expensive for many Indian researchers without institutional funding.

0
Sanjay Kumar Growing 145 pts · Accepted answer

I'd push back slightly on the idea that industry collaboration is universally beneficial for academic research. The incentive misalignment is real.

Industry partners want deliverables on a schedule. Academic research is often exploratory and unpredictable. When the two clash, it is usually the researcher who has to compromise.either by rushing results, constraining publication, or steering away from results that are inconvenient for the sponsor.

This doesn't mean industry collaboration is bad. But the terms matter enormously. IP rights, publication rights, and the ability to pursue negative results should be negotiated upfront and protected in writing.

0
Replying to Sanjay Kumar
Sanjay Kumar Growing 145 pts · Accepted answer

The IEEE TPAMI turnaround in my case was 7 months for the first review. Plan accordingly.

0
Bhavana Reddy Starting 30 pts · Accepted answer

Check whether your institution has a Springer/Elsevier waiver agreement.many NITs and IITs do.

0
Rajesh Kumar · Accepted answer

Overleaf + GitHub integration is underrated. You get version control and collaboration in one place.

0
Replying to Rajesh Kumar
Kiran Yadav Growing 130 pts · Accepted answer

Just to make sure I understand the Zotero workflow correctly.is the .bib file synced to Overleaf automatically every time you add a new paper, or do you need to trigger an export manually?

0
Jagdish Rawat · Accepted answer

Check whether your institution has a Springer/Elsevier waiver agreement.many NITs and IITs do.

0
Bhavana Reddy Starting 30 pts · Accepted answer

I found that writing a paper abstract first (before the paper) and then reverse-engineering the paper from the abstract helped with clarity enormously.

0
Revathi Chandrasekaran Active 410 pts · Accepted answer

I have successfully received PMRF (lateral entry, second year). Happy to share what worked in my proposal.

On the research proposal:

  • 60–70% technical approach, 30% context and impact
  • Preliminary results absolutely help. Coursework projects are fine if genuinely relevant
  • The societal impact section should be substantive.PMRF is a national initiative; reviewers want to see why this matters

On citations: Yes, cite your MTech thesis. Framing your PhD as an extension of prior work you did shows continuity of thought.

0
Replying to Revathi Chandrasekaran
Ganesh Menon Growing 140 pts · Accepted answer

I am in a very similar situation. Would you be willing to share the outline of your PMRF proposal? Not the content.just the section headings and approximate word allocation.

0
Admin NITK · Accepted answer

Highly recommend checking out the PMRF portal for the actual numbers.they update the stipend structure annually.

0
Replying to Admin NITK
Hemant Patwa · Accepted answer

Thank you! The reference to MuRIL is particularly useful.I had not considered it as an alternative to XLM-R for Indic languages.

0
Replying to Admin NITK
Divya Krishnan Growing 85 pts · Accepted answer

+1 to everything said above. My experience was identical.

0
Replying to Admin NITK
Jaya Lakshmanan Growing 50 pts · Accepted answer

This question comes up a lot. The answer really depends on your specific field and what your committee values.

0
Preethi Anand Distinguished 690 pts · Accepted answer

CSIR-UGC NET preparation: past papers are the most important resource. Everything else is secondary.

0
Replying to Preethi Anand
Nandita Ghosh Distinguished 720 pts · Accepted answer

This is a known issue with XLM-R on code-mixed data. The MuRIL preprint has a section specifically comparing these.

0
Prakash Iyer Distinguished 640 pts · Accepted answer

The IEEE TPAMI turnaround in my case was 7 months for the first review. Plan accordingly.

0
Replying to Prakash Iyer
Quamar Ahmed Growing 65 pts · Accepted answer

This is really helpful context. One more question: for the ethics committee approval, does the institution where the data is collected or the institution where the analysis is performed need to be the primary ethics approver?

0
Replying to Prakash Iyer
Prakash Iyer Distinguished 640 pts · Accepted answer

This is really helpful context. One more question: for the ethics committee approval, does the institution where the data is collected or the institution where the analysis is performed need to be the primary ethics approver?

0
Replying to Prakash Iyer
Indira Balan · Accepted answer

Just to make sure I understand the Zotero workflow correctly.is the .bib file synced to Overleaf automatically every time you add a new paper, or do you need to trigger an export manually?

0
Ravi Patel Starting 25 pts · Accepted answer

CSIR-UGC NET preparation: past papers are the most important resource. Everything else is secondary.

0
Replying to Ravi Patel
Ganesh Menon Growing 140 pts · Accepted answer

Really appreciate you taking the time to write this out in detail. This is going straight into my research notes.

0
Replying to Ravi Patel
Disha Malhotra Starting 8 pts · Accepted answer

I am in a very similar situation. Would you be willing to share the outline of your PMRF proposal? Not the content.just the section headings and approximate word allocation.

0
Replying to Ravi Patel
Sanjay Kumar Growing 145 pts · Accepted answer

Seconding the recommendation for Zotero. Game changer for managing references across multiple projects.

0
Quamar Ahmed Growing 65 pts · Accepted answer

Seconding the recommendation for Zotero. Game changer for managing references across multiple projects.