So even though it worked just fine all during play testing(grumble grumble) someone brought it to my attention that the pool may have some issues. I checked it in my game again and sure enough... problems!
The fix is simple though. Simply move the ladder over beside the diving board as shown.
This should solve the issue. Next time I do a curved pool I will remember not to put the ladder on the curve anymore. Sorry for the issue.
Thanks to the anon who let me know about this. I appreciate it!