**Official Statement From UltimateBet**
Tuesday, December 23, 2008 - COO
On Saturday, December 20, 2008 we were alerted to a problem with a hand involving PHILHELLMUTH and DOUBLEBALLER.
We realize that this issue is of great concern to UB’ers and the poker community. Since we learned of the problem we have worked tirelessly investigating and fixing the software issue that was responsible for the incorrect payout. We have now released an official statement, which you can read it on our corporate site tokwiro.com.
I became aware of the problem on Saturday evening, shortly before I made my original post on this blog. I must admit that the information on twoplustwo was very helpful to me in getting up to speed right away.
I know many of you wanted information immediately after the incident happened. I did make a post on the UB blog to let everyone know we were aware of the problem and the seriousness of it. At the time I did not comment further because I wanted to make sure that we had all the facts before we made any public statements. I would hate to say something that is inaccurate or even false because I rushed it out before understanding the problem completely.
The problem is truly and honestly just a software malfunction. I do realize this issue caused a great deal of concern because Phil Hellmuth, a spokesperson for the site, benefited from the malfunction and that is quite the coincidence. For this reason we are providing the public with as much information as possible and hope that by doing so, and answering as many of your questions as possible, you will be reassured that our site is a safe place to play.
The Issue:
As we explained in our press release, the incorrect payout was caused by a software malfunction. The software malfunction occurred when the winning player was disconnected from the hand at the precise millisecond the software was determining whom to award the prize. This in conjunction with the “player’s state” data being cleared from memory caused this extremely improbable incident to occur.
Originally we were unable to reproduce the error in our test environment, until we programatically forced a disconnect to happen at the precise millisecond the hand was being awarded.
We have never had an issue like this reported previously and we have reviewed all of our database and application logs for December and November and have been unable to find any historical occurrences. We are continuing to go back further into the historical data to ensure no hands from the past need to be corrected. Analyzing this data takes time but we will continue to go back further and further into our history to ensure no one else was affected.
The second, related issue:
During the weekend we released an update to our game servers which included new business logic to prevent the first incident from ever happening again. The new update unfortunately created a new logic issue resulting in 36 hands with incorrect payouts. The 36 hands affected were lower limit razz, stud, and hold’em hands. Upon learning of this problem, we were able to addressed it quickly and the affected players were refunded immediately. We are extremely confident that everything has been corrected and order has been restored but we’re continuing to monitor the system and are performing extensive tests to ensure there are no incorrect payout or disconnect issues in the future.
An unrelated issue:
Another player made a comment on our blog (Mark) and emailed our support team about a tourney (2120428) where 700 chips were mysteriously deducted from his stack mid tourney. What happened was his chip stack was reset to his starting chip amount (3000 chips). This happened because he won a satellite into the very tourney he was already playing in . This uncovered the unrelated problem, where the system reset his chip stack to the starting amount. This software issue was addressed immediately and Mark’s account was refunded.
Our company takes a lot of pride in our work and i think it’s safe to say we are embarrassed by these issues. We have an amazing software development team who have accomplished so much this year. I assure you they take this very seriously and I’m confident they have dealt with these issues professionally and I do not expect anymore problems to occur.
Our company is committed to being the most transparent site on the net and to providing our players with a personal touch that other sites do not. I speak to many players myself, I grant a lot of interviews, we offer phone support and our security team speaks with players on a daily basis. We hope that you start to notice the improvements we are continually making to improve our service and transparency.
I will try to blog and respond to your comments/questions the best I can over the holidays.
Happy holidays everyone!
Cheers,
Paul Leggett
On Saturday, December 20, 2008 we were alerted to a problem with a hand involving PHILHELLMUTH and DOUBLEBALLER.
We realize that this issue is of great concern to UB’ers and the poker community. Since we learned of the problem we have worked tirelessly investigating and fixing the software issue that was responsible for the incorrect payout. We have now released an official statement, which you can read it on our corporate site tokwiro.com.
I became aware of the problem on Saturday evening, shortly before I made my original post on this blog. I must admit that the information on twoplustwo was very helpful to me in getting up to speed right away.
I know many of you wanted information immediately after the incident happened. I did make a post on the UB blog to let everyone know we were aware of the problem and the seriousness of it. At the time I did not comment further because I wanted to make sure that we had all the facts before we made any public statements. I would hate to say something that is inaccurate or even false because I rushed it out before understanding the problem completely.
The problem is truly and honestly just a software malfunction. I do realize this issue caused a great deal of concern because Phil Hellmuth, a spokesperson for the site, benefited from the malfunction and that is quite the coincidence. For this reason we are providing the public with as much information as possible and hope that by doing so, and answering as many of your questions as possible, you will be reassured that our site is a safe place to play.
The Issue:
As we explained in our press release, the incorrect payout was caused by a software malfunction. The software malfunction occurred when the winning player was disconnected from the hand at the precise millisecond the software was determining whom to award the prize. This in conjunction with the “player’s state” data being cleared from memory caused this extremely improbable incident to occur.
Originally we were unable to reproduce the error in our test environment, until we programatically forced a disconnect to happen at the precise millisecond the hand was being awarded.
We have never had an issue like this reported previously and we have reviewed all of our database and application logs for December and November and have been unable to find any historical occurrences. We are continuing to go back further into the historical data to ensure no hands from the past need to be corrected. Analyzing this data takes time but we will continue to go back further and further into our history to ensure no one else was affected.
The second, related issue:
During the weekend we released an update to our game servers which included new business logic to prevent the first incident from ever happening again. The new update unfortunately created a new logic issue resulting in 36 hands with incorrect payouts. The 36 hands affected were lower limit razz, stud, and hold’em hands. Upon learning of this problem, we were able to addressed it quickly and the affected players were refunded immediately. We are extremely confident that everything has been corrected and order has been restored but we’re continuing to monitor the system and are performing extensive tests to ensure there are no incorrect payout or disconnect issues in the future.
An unrelated issue:
Another player made a comment on our blog (Mark) and emailed our support team about a tourney (2120428) where 700 chips were mysteriously deducted from his stack mid tourney. What happened was his chip stack was reset to his starting chip amount (3000 chips). This happened because he won a satellite into the very tourney he was already playing in . This uncovered the unrelated problem, where the system reset his chip stack to the starting amount. This software issue was addressed immediately and Mark’s account was refunded.
Our company takes a lot of pride in our work and i think it’s safe to say we are embarrassed by these issues. We have an amazing software development team who have accomplished so much this year. I assure you they take this very seriously and I’m confident they have dealt with these issues professionally and I do not expect anymore problems to occur.
Our company is committed to being the most transparent site on the net and to providing our players with a personal touch that other sites do not. I speak to many players myself, I grant a lot of interviews, we offer phone support and our security team speaks with players on a daily basis. We hope that you start to notice the improvements we are continually making to improve our service and transparency.
I will try to blog and respond to your comments/questions the best I can over the holidays.
Happy holidays everyone!
Cheers,
Paul Leggett