Maybe it would be an idea to instead of converting both to a set and doing set operations, just loop through ignore and do a tags.remove()?

The conversion to set is really fast, I suspect faster than looping through the list

For converting an iterable to a set it will need to loop through the iterable :).
Also, for converting it will check whether it already added the key previously for every element it's adding.
And given that it will also check for unicode versions of strings etc, I'm going to say that the conversion to a Set (which you do twice), is going to take longer.
Source:
https://github.com/python/cpython/blob/78d05eb847c6b8fede08ca74bb59210c00e4c599/Objects/setobject.c#L159

I don't convert twice, only once :)

You convert two lists :).

set(tags), set(ignore)

Updated

Looks good to me