The only way to really know if it's time to refinance is to do the math. Typically, the cost to refinance will be about 1-2% of the loans value. If you plan to stay in the house a long time (at least 2 years) and you can bring your interest rate down by 1% or more, it's probably worth it.
Here is a handy calculator from the Bank of America that you can use to help you make a decision.