Preheat the oven to 450°F with a rack in the center position. Grease an 8x8-inch baking dish with 1 tablespoon olive oil.
Add the potatoes and toss with 1 tablespoon of the oil, 1 teaspoon of the salt, and the black pepper. Roast until tender, 20-25 minutes.
Meanwhile, heat the butter and remaining 1 tablespoon oil in a medium saucepan over medium heat. Once the butter has melted add the shallot and cook, stirring, until tender, about 5 minutes. Add the garlic and cook until fragrant, about 1 minute more.
Add the cream, stock, rosemary sprigs, lemon zest, lemon juice, mustard, cayenne, and capers. Increase the heat to medium-high and bring to a simmer. Reduce the heat to medium low and cook until it begins to thicken, about 5 minutes.
Sprinkle the cod all over with the remaining 1 teaspoon salt.
Remove the potatoes from the oven, nestle in the fish and pour the sauce over the top. Bake uncovered until the fish is flaky, another 10-15 minutes.
Sprinkle with parsley, if using, before serving.