I managed to get it working!
this is the minimal file that works for me right now:
Type,Id,Parent Id,Name,Scope,Audience,Action Type,Sub Type,Text
Format Version,,,6.0,,,,
Customer List,<audience_id>,<customer_id>,,Customer,,Add,,
Customer List Item,,<audience_id>,,,,,Email,<hashed_email>
The Customer List line was missing. When Action Type is Add or Replace, the emails are added (Replace means all previous emails are dropped first). If it's Remove, the emails are removed.