Transfer olives to medium ceramic or terra cotta serving dish; set aside.
Using vegetable peeler, remove half of peel from lemon (yellow part only). Add lemon peel, olive oil, sherry vinegar, garlic, thyme, bay leaf and crushed red peppers to small skillet over medium heat. Cook, stirring occasionally, until oil begins to bubble, about 5 minutes. Pour warm oil mixture over olives; stir to combine. Let olives cool to room temperature; cover and refrigerate at least 2 hours, or up to 5 days.
Stir almonds into olive mixture, if desired; serve at room temperature.