Adjust oven rack to middle position and preheat to 325 degrees F. Spray 9x13-inch pan with non-stick cooking spray; set aside.
Cook bacon in large pan over medium heat until chewy, but not crispy, about 4-6 minutes. Move to paper towel-lined plate to drain until ready to use.
Place potatoes in large pot with cold water to cover by 1 inch. Bring to boil over high heat, add 1 teaspoon salt, and reduce heat to medium-low, simmer until potatoes are fork-tender (but NOT mushy), 10-14 minutes.
While potatoes boil whisk together mayonnaise, sour cream, milk, pickle juice, pickle, mustard, pepper, and celery seed in a large bowl. Once potatoes are done, drain and add them to bowl along with olives and cheese. Fold until potatoes are coated evenly.
Pour potato mixture into prepared pan and top with bacon. Bake for 45-55 minutes, or until bubbly and bacon is crisp. Serve immediately.
Store leftovers in the refrigerator for up to 4 days.