8011944: Sort fails with ArrayIndexOutOfBoundsExceptionSummary: Increase the size of pending stack and add test casesReviewed-by: alanb