It's fairly safe. You can leave dairy products unrefrigerated for an uncomfortable amount of time :) Butter, in particular, can last for days outside a fridge.
The bacteria that tends to infest dairy products will usually (but not always) turn it into something tasty like yogurt.
Don't get me wrong, you can definitely get sick from spoiled dairy products, but it's not a 100% thing.