I've never stayed in Whitby itself, but I've stayed at the Raithwaite Sandsend and would highly recommend it, it has everything you could want and within walking distance of Whitby.
Again not in the town but not too far away is https://howe-end.co.uk, a friend with young family has stayed there and rated that.
Alex (Watches of Whitby) rates https://sneatoncastle.co.uk
I also do some freelance work for Mulgrave, their estate has loads of self-catering cottages just north of the Town. https://mulgrave-estate.co.uk/cottages/