METHODS AND SYSTEMS FOR PROVIDING CUSTOM CRAWL-TIME METADATA

A method for providing metadata to a search engine for a document that is not in a mark-up language receives a request for contents of the document and locates metadata associated with the document. The method further creates name-value pairs for the metadata and provides to the search engine server a response comprising the name-value pair in an HTTP (or HTTPS) header and the contents of the document. In other implementations, a method includes sending a request for contents of the document and receiving a response to the request comprising an HTTP header with metadata about the document in a name-value pair and the document's content. The method also includes extracting the name-value pair from the HTTP header, creating a mark-up language tag for the name-value pair, and providing the make-up language tag and the contents of the document in a mark-up language format to a search index creation component.