Struct with dynamic sized array member

An external interface is sending over the wire basically a variable
sized struct, similar to:

struct ExternalMessage
{
  int m_numberOfOranges;
  int m_numberOfApples;
  Apple m_foo[=m_numberOfApples];
  Orange m_orange[=m_numberOfOrange];
};

Having to access this struct which is of unknown size until it is
received is not straightforward.

I was considering trying to use templates to pre-create every possible
struct layout and use the accessors that way.

Something like:

class VarLenBase {};

template<int NUM_ORANGE,int NUM_APPLE>
struct VarLenMsg : public VarLenBase
{
  int m_numberOfOranges;
  int m_numberOfApples;
  Apple m_foo[NUM_APPLE];
  Orange m_orange[NUM_ORANGE];

  // then accessors such as:
  Orange getOrange(int index) { return m_orange[index]; }
  // would just work without counting bytes and hopping
};

To pre-create them, i need some sort of mapping of the known params to
the concrete struct that matches:
  std::map<std::pair<int,int>, VarLenBase*> ;

But I'm strugglnig how to then use this information to cast the
received buffer as the type that I now know it is.

Is there a better way (probably yes). using std::vectors is not the
answer since i am not in control of the sender, and it has to be sent
over a network anyway.
Thanks,
JC