It costs ~£15 to start a limited company in England and Wales (a separate jurisdiction to Scotland and Northern Ireland). So long as you keep accurate records and file your annual accounts correctly, that's all there is. An accountant would normally do the filing for you. For a small business this can often be done for £500-£1000. If you do your own bookkeeping and keep accurate records, that will keep costs down.
As your business will be domiciled in the UK, it will be subject to the UK's tax regime. But if you're outside the UK, your salary from the business shouldn't be subject to UK taxation (do check that, though - double-taxation isn't something I know much about).
TL;DR - Find an accountant in the UK who'll be willing to give you a free consultation (most of them). You'll probably find one on ukbusinessforums.co.uk (it's where I found mine).