The MP and former minister, in his fifties, whose identity cannot be disclosed for legal reasons, was held on Saturday over allegations of rape, sexual assault and coercive control.
He has been released on bail until the middle of this month as police continue an investigation.
The Conservative Party has said they will take no action against him while the allegations are investigated.
With @The_TUC @fawcettsociety @womensaid @RapeCrisisEandW @EVAWuk we're calling for the Conservative MP suspected of rape to be suspended while a full investigation is carried out. Parliament is no different from other workplaces & staff must be protected.https://t.co/Pg0Mtc6zsx— Centenary Action Group (@CentenaryAction)August 4, 2020
The TUC, Women’s Aid, the Fawcett Society, the Centenary Action Group and the biggest trade unions representing staff in parliament - Prospect, Unite, PCS and the FDA - released a joint statement on the party's decision.
They said: “Given the seriousness of the allegations we’re calling for the MP to be suspended pending a full investigation.
“The House of Commons should be no different from any other workplace. Safety and the safeguarding of staff must be paramount.
"Failure to suspend is yet another example of minimising violence against women and girls and failure to believe victims and survivors when they bravely speak out.”
The groups also said they do not feel confident that Westminster is "a safe and equal place for women to work".
As well as calling for the MP to have the whip suspended, the group of campaigners wants political parties to make their systems for dealing with complaints publicly available.
This would include specific protocols and training for whips on how to respond to disclosures appropriately.
On Monday, Anne Milton, the Tories’ former deputy chief whip, urged the accused MP to give up the whip voluntarily while the case is resolved, and called on the Conservative Party to overhaul its disciplinary process.
She spoke out after the alleged victim told The Times she was devastated to learn that the MP would continue as a full member of the party .
The former researcher, who cannot be identified, says that she was assaulted four times between July 2019 and January this year, including claims of a rape necessitating hospital treatment.
Mark Spencer, the chief whip, said on Monday that the government took the matter “very seriously” but it was for the police to investigate.
Speaking in a clip on Sky News, Mr Spencer said: "They are very serious allegations and we do take those allegations very seriously.
"I think it is down to the police to do that thorough investigation, not for the Whips Office to investigate this alleged crime, it is for the police and the authorities to do that.
"Once they've come to that conclusion, then we can assess where we're at and the position that the MP find themselves in.
"Of course, we've got to bear in mind the victim, we don't want to do anything to identify the victim at the same time."
Last week, The Times reported that the woman had raised her allegations with Mr Spencer in April but the chief whip did not take any action against the MP and did not encourage the woman to contact the police.