And CBP does have the right to ask who you are and where you're going when you cross the border. It is their business, and has been since the 18th century.
EDIT: I should clarify because I'm being somewhat glib. First, 4th amendment does not apply to routine border searches, and it was the First Congress, containing many of the framers, that instituted routine border searches: http://blog.cybersecuritylaw.us/2013/09/23/david-house-dhs-a....
Second, while the Supreme Court has not ruled on the issue, the various circuits have held that 5th amendment does apply, but Miranda warnings are not required: http://fourthamendment.com/blog/index.php?blog=1&title=e_d_n....
These are both rooted in the long-standing right of sovereign entities to defend their borders.
Third, I'm not aware of any Supreme Court case that says you can't be compelled to answer questions at the border. Currently, there is simply no law compelling you to answer questions, but it's not clear such a law would be unconstitutional if it existed. So there is nothing wrong with CBP asking, and nothing wrong with you refusing to answer, and nothing wrong with CBP giving you extra scrutiny, within the limits of the 4th amendment, for refusing to answer the questions.
Fourth, I'm in the camp that thinks Miranda shouldn't even apply to questioning at the border. See: http://www.volokh.com/2013/06/17/do-you-have-a-right-to-rema.... The 5th amendment's text is simple: "No person... shall be compelled in any criminal case to be a witness against himself." The historical practice and plain meaning of this text was, before the 1960's, that you couldn't force a defendant to testify against himself in a criminal proceeding. This was a specific response to the practice of the Star Chamber in England. It's a pretty big stretch to say that this should limit the police's ability to ask you questions, especially at the border where the state's interest in security is relatively stronger.