I'm a bit of hacker. I've worked with serious software engineers, and I'm definitely not one of those, although I can be when it's required. They're the types that spend more time writing and talking about what they're going to code than they do actually coding it. The word engineer suits these types perfectly. A piece of software to them is constructed from piece by piece of pre-defined, pre-designed and pre-fabricated code. There is more art than science to what I do. I liken what I do to gardening more than building a bridge. To me, software is more of an organic thing and I start software the way I start in the garden. I pick up a shovel and start digging. Over time as plants grow (or die), the yard changes and you change with it. I'm not building bank software or landing planes. I'm building to an ever changing target, evolving my software to suit a need that is never well defined, never fully known. It's whatever I feel like making it.
The big disclaimer here is that sometimes I need to be an engineer. If I'm working for a client on a fixed price project, then obviously everything needs to be engineered to a certain goal. So I can work that way, I just prefer not to. My client work is structured and defined, my pet projects are the result of an ever evolving process of discovery and learning. Of course, that isn't to say you can forget about security, performance or reliability...but it means these things are just one part of an iterative process, not a project in themselves.
For the most part, not being constrained by things like "best practices" is a wonderful freedom. I think of something, and I start coding it. When I'm done, I might go back and tweak or even throw away the prototype and start again (kind of like how we're onto our second Japanese Maple, after the first one died). It's wonderful when you know nobody will ever look at your code, or discover your hacks, it allows you to solve problems now without worrying too much about later. It's like an author taking short cuts with facts, names and places instead of wasting time researching. It just gets in the way of writing the story. But there is a big problem with this type of development. What you gain in agility, you lose in portability. By coding for yourself, you're forgetting about everyone else.
If you build a successful product, there is a fair chance that at some point you're going to need to get someone else involved. It may be as an employee joining your coding team, or showing a potential investor what you've been working on or it may be by releasing an API for others to integrate with your system. For the first time you're suddenly faced with that embarrassing feeling of your parents dropping in to visit...and it's the morning after a really really big party. It's at this point you suddenly regret all the shortcuts, the laziness, the hacks and the sambuca...oh Lord the sambuca!
That's where writing a public API from day 1 comes in.
You don't have to have your whole house neat, just the bits your parents are going to be seeing. Writing a public API as you develop your system forces you to keep a certain level of decency without constraining your ability to be agile elsewhere. By using your own API you also force yourself to follow the same rules that you expect others to follow when interfacing with your code. It prevents you taking shortcuts that you rightly wouldn't let anyone else take.
If you're an agile or "lean" developer, the one thing you never think about is what this app is going to need to do in the future. You're only ever worried about what it needs to do today. This means your code can evolve into a horrible flying spaghetti monster (if you believe in such things). What makes sense today, may not seem like such a good idea tomorrow. By having a well defined core API you have at least one part of your system (hopefully the important bit) that is well documented, well considered and well written. It's something akin to maintaining a tidy formal lounge whilst the rest of your house is being subjected to an ill-considered conga line.
When it eventually comes time to make your API public, it's already tested and known to work. You know what problems your users are likely to encounter, because you've already encountered them yourself. Best of all, you're not trying to shoehorn a heap of public points of access where they were never intended. Writing an API as you go means you solve a lot of problems before they ever happen...and all without having to think about it too much. The API is as flexible as you want it to be until the day you make it public.
As you grow your application, your API grows with it. The more reliable it becomes, the more likely you will turn to it rather than hacking in a new piece of code elsewhere. When new developers come on board they have something they can immediately recognise and understand. We're all terrible at documenting our code, but with a published API you all of a sudden are advertising what you're application does...you want it to be well documented. Your users won't stand for anything less.
All of this is the main reason most developers recommend working on open source projects. Open source means everything you do is available for anyone to see. Hacks and cheats are found and highlighted by others, and hopefully you'll learn a better way in the process. You lose this benefit of peer review when you're working on your own private projects. Writing to your own API is the next best thing. Hacks look a whole lot worse when you have to document them for someone else.