r/research • u/No-Resolution-9253 • 5d ago

My ANOVA test results are non-significant (p sizes as 0.7 and 0.3) for a sample of 310 participants. Will a scopus indexed journal accept such a paper? (Q1/Q2)

My thesis paper is on the topic of whether the labels given above differently generated logos; one created by humans and the logo colour modified and picked by AI - ( ONLY colour palette variations). My tests were a form with true labels, another test b with labels switched ( false labels ) and another with control group ( no labels )

With a sample size of 310, I did a one way ANOVA test and the results are very non-significant. I am required to publish the paper in a scopus indexed journal Q1/Q2. Do journals like that accept non significant results?

0 Upvotes

33% Upvoted

23 comments sorted by

15

u/Magdaki Professor 5d ago

Is this from the data you gathered by recruiting on Reddit that I can see in your profile? If so, then the data quality is likely *very* low, which could explain the results. You should consider a better data gathering method. To answer your question, right or wrong, it is challenging to publish insignificant results. The problem is it doesn't really contribute much so it can be hard to convince reviewers that it adds to the body of human knowledge.

13

u/Tall_Monk_7306 5d ago

p-hacking your way out of a null result with a sample that size is a trap. reviewers can smell it and they'll tear it apart

the bigger issue is the reddit recruitment angle. that data is basically noise, no journal worth publishing in will take it seriously

4

u/Kanoncyn Graduate Student 5d ago

Both are an issue. You can’t conduct an ANOVA with 31 people and expect something significant unless your etas are massive, AND you have a homogenous sample that minimizes error. You also can’t expect a reasonable set of responses recruiting from Reddit the way OP does. I’d say this work is by definition unpublishable because it’s bad science, and these, plus 10+ other issues I can list, are the reasons why. A null result with good power is always publishable with the right framing. 

1

u/Aggressive_Roof488 5d ago

Or worse, the reviewers won't care and it's published.

2

u/Kangouwou 5d ago

That's interesting. Obviously, data collection is central for a proper analysis, but what do you envision as a better way of collecting data ? For instance, what is OP randomly stopped people on the street to make that test ? Do you think it would be better that recruiting people on Reddit ?

6

u/Magdaki Professor 5d ago

Recruiting on Reddit has two main problems.

First, you cannot really confirm your population. All you can really say is that it is a population of Reddit users, which usually isn't not suitable for publication because there are baked in biases among people who use Reddit. You maybe can get away with it under some conditions, but it is a problem.

The second problem I would say is the bigger problem though. A lot of the survey subreddits are exchanges. So what happens is somebody wants their survey filled out, so they will quickly fill out yours and then ask (and foolishly expect you) to fill theirs out with care. Obviously, this won't be everybody but it is enough that you cannot use the data.

1

u/MammothComposer7176 5d ago

You usually pay participant using serious platforms. Such as Prolific.

1

u/No-Resolution-9253 5d ago

I got a few from reddit, most of mine are offline sampling, its 310 participants not 31

6

u/Magdaki Professor 5d ago

You should probably remove all the ones from Reddit. If that doesn't improve the analysis, then you may need to look at your instrument or simply accept that there is no significant effect.

0

u/No-Resolution-9253 5d ago

I think theres around 20 from reddit - a lot of people comment done but dont really fill the surveys so it shouldn't be hard to remove. but thank you very much.

3

u/Magdaki Professor 5d ago

Good luck! Unfortunately, I doubt removing 20 is going to make a big difference, so as above I think you need to look at the instrument you're using. And if you still find it is valid then you may need to revisit the research from the start.

1

u/No-Resolution-9253 5d ago edited 5d ago

ive used two psychometric assessments that are standardised so I doubt thats the issue :(.

The aim of the paper was to revisit the idea of "AI aversion" that is present in a lot of the current literature.

Technically it does contradict the AI aversion; so maybe I can make something of it just yet? I'll have to speak to my guide, lets see what can be done. Thank you for your help

1

u/Magdaki Professor 5d ago

I hope it all works out. :(

1

u/Minimum_City_8387 5d ago

You can choose to publish a null results on scientific reports or somethings BUT 31 is extremely low. Obviously it depends on many things, there are some instances where 31 is considered to be okay

1

u/No-Resolution-9253 5d ago

its 310 my badddd

1

u/prof_mcquack 5d ago

Negative results are hard to publish unless you’re testing a hypothesis that has a lot of support or buzz around it. Add that to a small sample size (is it 31 or 310?) and yeah it’s gonna be a challenge. That said you can probably find a few high profile journals that will gargle your metaphorical balls for doing a paper about AI in any way. 

1

u/EngineeringNew7272 5d ago

The p-value matters less than the knowledge the science community gains from it.
Your study doesn’t seem to answer a genuine question.

Think of a large study testing whether the MMR vaccine is linked to autism: if the result was non-significant, this would still tell a lot.

1

u/vaelux 5d ago

Pre-register your studies and you won't have to worry about such things.

1

u/ForeignAdvantage5198 5d ago

it depends on what the paper says and your justifcation

1

u/cloudian0 3d ago

Yes, Q1/Q2 journals can publish non-significant results. The key is how you interpret them. With 310 participants, I'd focus on reporting effect sizes and confidence intervals rather than just p-values. A non-significant result doesn't necessarily mean there's no effect. If the confidence intervals are narrow enough, you might actually have a strong argument that AI-generated and human-designed logos don't differ much in how people perceive them. That could be an interesting finding on its own.

1

u/No_Young_2344 1d ago

Is the ANOVA the only thing for the research? Then I think it might hard. Do you have other analysis, like regression, qualitative analysis?