Reggae artist Hector Delgado and rapper Jayson Musson say the producer of the chart-topping hit Harlem Shake did not have permission to use their voices.
The pair are now asking for compensation from record label Mad Decent Records, which released the single, reports the New York Times.
Harlem Shake has become an internet sensation on video website YouTube.
Producer Harry Rodrigues, also known as DJ Baauer, and Mad Decent Records declined to comment.
Musson told the New York Times the record label had been "more than co-operative" but an agreement had not yet been reached.
The song has inspired thousands of people around the world to upload videos of themselves dancing along to the first 30 seconds of the track - up to 4,000 a day are currently being put onto YouTube, according to the website.
The song was initially released in 2012.
Delgado can be heard at the beginning of the track singing, "Con los terroristas," a sample from a single he released in 2006.
Musson shouts, "Do the Harlem Shake," about 15 seconds in, which he says is taken from a rap by his group Plastic Little in 2001.
In the New York Times, Delgado's agent, Javier Gomez, described the situation as "a clear breaking of intellectual property rights".
Last month in an "ask me anything" session on website Reddit, Rodrigues was asked about the origin of a female Spanish vocal that also features on the track.
"Found it on the innerweb," he wrote in reply.
Barney Hooper, from PRS for Music, which represents the rights of music creators, told the BBC: "If one piece of music is sampled in another, then typically the creators of the original work would be entitled to a share of royalties when the new work is played, performed or reproduced," he said.
"A song can have a number of songwriters/composers and use samples of other works.
"If this is the case, all could have an ownership share in the new work and these would be registered with organisations like PRS for Music.
"We then pay out the royalties we collect based on the ownership shares registered with us."
However the situation was sometimes complicated, Mr Hooper added.
"There are often ownership disputes relating to popular works where others musical works are sampled," he said.