Check to see what was signed. There is a lot of legislation now around privacy and personal and confidential information. There are two avenues, the perspective employer must obtain written permission to contact the references (and must also have something explicit to say they can contact contact references off the list.)
The first step is to ask HR for a freedom of information request regarding who was contacted for a reference. If they state confidentiality, just ask for the names not the references, not what was discussed (for now anyway). Also ask for their corporate policy on the protection of personal information. Start pointing out where they have violated their own policy. Your goal is to make the company want to make you go away and the easiest way for them to do it is to give you the names.
If you are able to find out who gave the references, go to the HR of those companies and file a complaint that those employees released personal information about you without your authorization. It doesn't matter if the references are good or bad, unless they had authorization to release personal information they are not allowed to do so. The company also had a duty to inform you of any breach of confidential information so the fact that you found out and was not informed by them is a breach on their part.
It probably isn't worth perusing anything through the courts but you will accomplish a couple things. One, you will know who gave the poor references, Two, the individuals who gave the reference will by unlikely to do it again, and three, it will likely cause them some grief.