I have a copy of the resource database, so that I can search also code for system procedure. I found that this error is raised in sp_addarticle:
-- Get applock to co-ordinate with snapshot agent. Basically,
-- we want to prevent a new article from being added while a
-- snapshot is being generated.
EXEC @applock_result = sys.sp_getapplock @Resource = @publication,
@LockMode = N'Exclusive',
@LockOwner = N'Transaction',
@LockTimeout = 0,
@DbPrincipal = N'db_owner'
IF @applock_result < 0
BEGIN
RAISERROR(21854, 16, -1, @publication)
GOTO UNDO
END
It seems that there is another operation running on the publication which also has taken this application lock.
I don't whereelse this application lock may be taken, but one possibility would be to set the database offline and then online again to roll back all open transaction.
You could also look for processes with this application lock, but I'm not sure if it is worth the effort.