I haven't used Protovis in a Mendix application but I have done some visualization of metadata with raw canvas and RaphaelJS. Looking at Protovis I don't see any reason why you couldn't use it.
What is the source of the data you want to visualize? XPath queries? These result in lists of MxObjects and you can use the API to extract values from them.
The Reporting module works a little differently - you can't currently send arbitrary OQL queries to the server. You construct the queries in the Modeler and from the client you reference these queries with their identifiers (and optional parameters). AFAIK this module is not currently open for third-party development.