C++ OO design question

In a recent job interview, the interviewer asked me how I'd design
classes for the following problem: let's consider a hypothetical
firewall, which filters network packets by either IP address, port
number, or both. How should we design the classes to represent these
filters?

My answer was:

class FilterRule
{
public:
     virtual bool Accept(const Packet&) const = 0; //Packet is a
representation of a network packet
};
class FilterByIP: public FilterRule { /* members here */ };
class FilterByPort: public FilterRule { /* members here */ };

class Filter
{
public:
     bool Accept(const Packet&) const; //returns true if ALL
filterRules Accept() the Packet
private:
     std::vector<FilterRule> filterRules;
};

However, the interviewer said that he preferred this solution instead:

class Filter
{
public:
     virtual bool Accept(const Packet&) const = 0;
};
class FilterByIP: public Filter { /* members here */ }
class FilterByPort: public Filter { /* members here */ }
class FilterByIPAndPort: public Filter
{
protected:
     FilterByIP ipFilter;
     FilterByPort portFilter;
/* other members */
};

I reasoned that with his solution, there may be too many class numbers
if down the road we decide to filter packets by methods other than IP
address and Port, but somehow he was not convinced. Oh well, I didn't
get the job, but this question continues to haunt me to this day. What
do you C++ experts think? Or is there another better solution that I
did not consider? Thank you.