Hello,
Did you check that you’re using the right atoms for these two residues? You can find the list of the proper atoms for residues ACE and CTN there: http://milou.science.uu.nl/services/HADDOCK2.2/library.html
B1N is not part of modified amino-acids supported by HADDOCK so you’ll need to define it as HETATM in your PDB file and get its topology be defined by PRODRG during the HADDOCK run.
Edit: To be complete, ACE and CTN are two groups you’ll add complementary to the residues they are attached to, do not replace the whole residue with them.
I hope this will help,
Mikael