2 of my friends are doing RTW's (27,000 HARD miles so far via England, Europe, the Stans, Russia, Mongolia, Road of Bones, USA) on a 2004, and 2007 XR650L. Both bought used, and for great deals.
My suggestion. Search Craiglook.com, it is the entire USA craiglist system thrown into one awesome webpage. You will find very many xr650l's for great prices. If you can hold out and put down $3000 cash, My bet is that you can find a 2005-6 for that price, and likely with less then 8,000 miles.
California is JAM PACKED with used XR650L's. I once saw a 2007 with 6500 miles go for $2700 in Portland, Or. I search the site every day. You will have no problem finding one in California.
I would NOT buy a new one, but instead, would most definitely buy used. You can also apply for a bank loan that would allow you to finance a used bike sold via craigslist. That is another option. You do not have to be financed by the dealer.
Many people pay for these bikes outright or get 2-3 year loans and pay them off quickly but hardly ride the dang things. Then they want out, so they sell cheap. Thats what you are looking for, not a dealer priced dousing from the rip off hose.