Login | Register
My pages Projects Community openCollabNet

Discussions > users > [maxq-users] Patch: Adding Proxy support to MaxQ

maxq
Discussion topic

Back to topic list

[maxq-users] Patch: Adding Proxy support to MaxQ

Author fcohen
Full name Frank Cohen
Date 2003-09-23 11:05:12 PDT
Message A couple of TestMaker users asked me to add support for proxys to MaxQ so they could record tests through their corporate proxy server. I implemented proxy support for MaxQ by changing these files:

ProxyServer.java - now has a new init method to pass in the proxy name, number, and a flag determining if MaxQ should use the proxy

RequestHandler.java - if the proxy flag (set in ProxyServer's init method) is set, RequestHandler now opens the socket to the Proxy server, instead of the host

HttpRequestHeader.java - manipulates the HTTP header to send the request to the proxy

While testing this in my lab I noticed that every-other request was taking a long time (15 seconds +) to complete and close. I am going to look into this in the next day or two.

Also, the way I implemented this changes MaxQ to use HTTP 1.1 when the proxy option is enabled. This does not appear to have any effect on MaxQ working with the hosts I tested (Apache, Zope, and IIS Web servers) but this may be opening a big can of worms.

Lastly, this does not implement proxy authentication.

I'm open to any and all comments, feedback, criticism about this patch.

I am not sure what the contribution process is for MaxQ so I am just posting it here. I am willing to check it into the MaxQ cvs if needed. I would like to include this patch in the next release of TestMaker (which integrates MaxQ) so please let me know if you disagree with the patch. I'm willing to change the patch, rather than branching MaxQ.

-Frank Cohen
http://www.pushtotest.com/ptt


--------------------​--------------------​--------------------​--------------------​---
ProxyServer.java

package com.pushtotest.testm​aker.gui.NewAgentWiz​ard.AgentRecorder;

/**
 * Implementation of a generic proxy server. Creates a ProxyClient for every
 * connection it receives.
 *
 * This is largely based on MaxQ, an open-source test utility for automatically
 * writing test scripts in Jython. Details at: http://maxq.tigris.org
 *
 * For more info check http://www.pushtotest.com or send email to info at pushtotest dot com
 * This source code is licensed under terms described in the License.txt file.
 *
 */

import java.io.*;
import java.net.*;
import java.util.*;

public class ProxyServer extends Thread {

    private Vector listeners;
    private ServerSocket srvSock;
    private int count;
    
    private String proxy = null;
    private int proxynum = 80;
    private boolean proxyflag = false;

    /**
     * Attempts to start a proxy server on port 0, the default,
     * if successful,it starts the proxy server on a new thread.
     */
    public ProxyServer() throws IOException {
        this(0, null, 0, false);
    }

    /**
     * Attempts to start a proxy server on the specified port.
     * if successful,it starts the proxy server on a new thread.
     */
    public ProxyServer( int port ) throws IOException
    {
        this(0, null, 0, false);
    }
    
    /**
     * Attempts to start a proxy server on the specified port,
     * plus this proxy uses a specified proxy.
     * If successful, it starts the proxy server on a new thread.
     *
     * @param theflag = true if the ProxyServer should use a proxy itself
     * @param theproxynum = the proxy port number to send requests to
     * @param theproxy = the proxy address
     *
     * Note: This version of ProxyServer does not support proxy authentication.
     * That will likely come in a future version.
     */
    public ProxyServer( int port, String theproxy, int theproxynum, boolean theflag ) throws IOException
    {
        proxyflag = theflag;
        proxy = theproxy;
        proxynum = theproxynum;

        srvSock = new ServerSocket(port);
        listeners = new Vector();
    }

    public void addObserver(ProxyObserver proxy) {
        listeners.addElement(proxy);
    }

    public void removeObserver(ProxyObserver proxy) {
        listeners.removeElement(proxy);
    }

    public void processRequest(HttpR​equestHeader header,
        byte[] requestBody, byte[] response)
        throws IOException {

        for(Enumeration e = listeners.elements(); e.hasMoreElements();) {
            ProxyObserver pl = (ProxyObserver)e.nextElement();
            pl.processRequest(header, requestBody, response);
        }
    }

    public void run() {
        while(true){
            try{
                Socket s = srvSock.accept();

                RequestHandler handler = new RequestHandler(this, s);

                Thread t = new Thread(handler, "MaxQRequestHandler #" + count);
                t.start();
                count++;
            }
            catch(Throwable t){
                t.printStackTrace();
            }
        }
    }

    int getLocalPort() {
        return srvSock.getLocalPort();
    }

    public String getProxy() { return proxy; }
    
    public int getProxyNum() { return proxynum; }
    
    public boolean getProxyFlag() { return proxyflag; }

}


--------------------​--------------------​--------------------​--------------------​---
HttpRequestHeader.java

package com.pushtotest.testm​aker.gui.NewAgentWiz​ard.AgentRecorder;

import java.io.InputStream;
import java.io.DataInputStream;
import java.util.StringTokenizer;

/**
 * Parses and stores a http server request. Originally posted to
 * comp.lang.java in 1996.
 *
 * @author Sherman Janes
 */
 public class HttpRequestHeader {
   
   /**
    * Http Request method. Such as get or post.
    */
   public String method = new String();

   /**
    * Http Host header value.
    */
   public String host = new String();
   
   /**
    * Http Connection header value.
    */
   public String connection = new String();
   
   /**
    * Http Proxy-Connection header value.
    */
   public String proxyconnection = new String();
   
   /**
    * The requested url. The universal resource locator that
    * hopefully uniquely describes the object or service the
    * client is requesting.
    */
   public String url = new String();

   /**
    * Version of http being used. Such as HTTP/1.0
    */
   public String version = new String();

   /**
    * The client's browser's name.
    */
   public String userAgent = new String();

   /**
    * The requesting documents that contained the url link.
    */
   public String referer = new String();

   /**
    * A internet address date of the remote copy.
    */
   public String ifModifiedSince = new String();

   /**
    * A list of mime types the client can accept.
    */
   public String accept = new String();

   /**
    * The clients authorization. Don't belive it.
    */

   public String authorization = new String();
   /**
    * The type of content following the request header.
    * Normally there is no content and this is blank, however
    * the post method usually does have a content and a content
    * length.
    */
   public String contentType = new String();
   /**
    * The length of the content following the header. Usually
    * blank.
    */
   public int contentLength = -1;
   /**
    * The content length of a remote copy of the requested object.
    */
   public int oldContentLength = -1;
   /**
    * Anything in the header that was unrecognized by this class.
    */
   public String unrecognized = new String();
   /**
    * Indicates that no cached versions of the requested object are
    * to be sent. Usually used to tell proxy not to send a cached copy.
    * This may also effect servers that are front end for data bases.
    */
   public boolean pragmaNoCache = false;

   static String CR ="\r\n";

   private boolean proxyFlag = false; // If set to true, then the request is going through another proxy
   
   public HttpRequestHeader()
   {
        this( false );
   }
   
   public HttpRequestHeader( boolean pflag )
   {
        this.proxyFlag = pflag;
   }
   

/**
 * Parses a http header from a stream.
 *
 * @param in The stream to parse.
 * @return true if parsing sucsessfull.
 */
public boolean parse(InputStream In)
   {
       String CR ="\r\n";

       host = "";
       
       /*
        * Read by lines
        */
       DataInputStream lines;
       StringTokenizer tz;
       try {
           lines = new DataInputStream(In);
           tz = new StringTokenizer(line​s.readLine());
       } catch (Exception e) {
           return false;
       }

       /*
        * HTTP COMMAND LINE < <METHOD==get> <URL> <HTTP_VERSION> >
        */
       method = getToken(tz).toUpperCase();
       url = getToken(tz);
       version= getToken(tz);
               
       while (true) {
           try {
               tz = new StringTokenizer(line​s.readLine());
           } catch (Exception e) {
               return false;
           }
           String Token = getToken(tz);
           
           // look for termination of HTTP command
           if (0 == Token.length())
               break;
           
           if (Token.equalsIgnoreC​ase("USER-AGENT:")) {
               // line =<User-Agent: <Agent Description>>
               userAgent = getRemainder(tz);
           } else if (Token.equalsIgnoreC​ase("ACCEPT:")) {
               // line=<Accept: <Type>/<Form>
               // examp: Accept image/jpeg
               accept += " " + getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("HOST:")) {
               // Host =<Host: <URL>>
               host = getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("CONNECTION:")) {
               // Connection =<Connection: keep-alive>
               connection = getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("PROXY-CONNECTIO​N:")) {
               // Proxy-Connection =<Proxy-Connection: keep-alive>
               proxyconnection = getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("REFERER:")) {
               // line =<Referer: <URL>>
               referer = getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("PRAGMA:")) {
               // Pragma: <no-cache>
               Token = getToken(tz);

               if (Token.equalsIgnoreC​ase("NO-CACHE"))
                   pragmaNoCache = true;
               else
                   unrecognized += "Pragma:" + Token + " "
                       +getRemainder(tz) +"\n";
           } else if (Token.equalsIgnoreC​ase("AUTHORIZATION:"​)) {
               // Authenticate: Basic UUENCODED
               authorization= getRemainder(tz);

           } else if (Token.equalsIgnoreC​ase("IF-MODIFIED-SIN​CE:")) {
               // line =<If-Modified-Since: <http date>
               // *** Conditional GET replaces HEAD method ***
               String str = getRemainder(tz);
              int index = str.indexOf(";");
              if (index == -1) {
                   ifModifiedSince =str;
               } else {
                   ifModifiedSince =str.substring(0,index);
                  
                  index = str.indexOf("=");
                  if (index != -1) {
                      str = str.substring(index+1);
                      oldContentLength =Integer.parseInt(str);
                  }
              }
           } else if (Token.equalsIgnoreC​ase("CONTENT-LENGTH:​")) {
               Token = getToken(tz);
               contentLength =Integer.parseInt(Token);
           } else if (Token.equalsIgnoreC​ase("CONTENT-TYPE:")​) {
               contentType = getRemainder(tz);
           } else {
               unrecognized += Token + " " + getRemainder(tz) + CR;
           }
       }
       return true;
   }
           
   /*
    * Rebuilds the header in a string
    * @return The header in a string.
    */
   public String toString(boolean sendUnknowen) {
       String Request;

       if (0 == method.length())
            method = "GET";

       Request = method + " ";

       if ( proxyFlag )
       {
            if ( url.toLowerCase().startsWith( "http" ) )
            {
                Request += url + " HTTP/1.1" + CR;
            }
            else
            {
                Request += "http://" + host + url + " HTTP/1.1" + CR;
            }

            Request +="Host: " + host + CR;

           if (0 < proxyconnection.length())
               Request +="Proxy-Connection: " + proxyconnection + CR;
       }
       else
       {
           Request += url + " HTTP/1.0" + CR;
       }
       
       if ( ( 0 < host.length() ) && ( ! proxyFlag ) )
           Request +="Host: " + host + CR;

       if ( ( 0 < connection.length() ) && ( ! proxyFlag ) )
           Request +="Connection: " + connection + CR;
           
       if (0 < userAgent.length())
           Request +="User-Agent: " + userAgent + CR;

       if (0 < referer.length())
           Request+= "Referer: "+ referer + CR;

       if (pragmaNoCache)
           Request+= "Pragma: no-cache" + CR;

       if (0 < ifModifiedSince.length())
           Request+= "If-Modified-Since: " + ifModifiedSince + CR;
           
       // ACCEPT TYPES //
       if (0 < accept.length())
           Request += "Accept: " + accept + CR;
       else
           Request += "Accept: */"+"* \r\n";
    
       if (0 < contentType.length())
           Request += "Content-Type: " + contentType + CR;

       if (0 < contentLength)
           Request += "Content-Length: " + contentLength + CR;
                           
       if (0 != authorization.length())
           Request += "Authorization: " + authorization + CR;

       if (sendUnknowen) {
           if (0 != unrecognized.length())
               Request += unrecognized;
       }
       
       Request += CR;

       return Request;
   }


   /**
    * (Re)builds the header in a string.
    *
    * @return The header in a string.
    */
   public String toString() {
       return toString(true);
   }

   /**
    * Returns the next token in a string
    *
    * @param tk String that is partially tokenized.
    * @return The remainder
    */
   String getToken(StringTokenizer tk){
       String str ="";
       if (tk.hasMoreTokens())
           str =tk.nextToken();
       return str;
   }
   
   /**
    * Returns the remainder of a tokenized string
    *
    * @param tk String that is partially tokenized.
    * @return The remainder
    */
   String getRemainder(StringTokenizer tk){
       String str ="";
       if (tk.hasMoreTokens())
           str =tk.nextToken();
       while (tk.hasMoreTokens()){
           str +=" " + tk.nextToken();
       }
       return str;
   }

}


--------------------​--------------------​--------------------​--------------------​---
RequestHandler.java


package com.pushtotest.testm​aker.gui.NewAgentWiz​ard.AgentRecorder;

import java.io.*;
import java.net.*;

/**
   handles the complete cycle for the communication between an http client and
   server
 *
 * This is largely based on MaxQ, an open-source test utility for automatically
 * writing test scripts in Jython. Details at: http://maxq.tigris.org
 *
 * For more info check http://www.pushtotest.com or send email to info at pushtotest dot com
 * This source code is licensed under terms described in the License.txt file.
 *
 */

public class RequestHandler implements Runnable {

    private InputStream clientIn, serverIn;
    private OutputStream clientOut, serverOut;

    private HttpRequestHeader header;

    private ProxyServer proxyServer;
    private Socket clientSocket;
    private ByteArrayOutputStream buffer;

    RequestHandler( ProxyServer proxyServer, Socket s )
    {
        clientSocket = s;
        buffer = new ByteArrayOutputStream();
        this.proxyServer = proxyServer;
    }

    /**
     * Gets the client information and establishes a connection with the
     * server. This method is blocking.
     */
    private boolean initClientServerConn​ections(Socket s) throws IOException
    {
        clientIn = s.getInputStream();
        clientOut = s.getOutputStream();
        
        header = new HttpRequestHeader( proxyServer.getProxyFlag() );
        header.parse( clientIn );

        if ( header.url == null || !header.url.startsWith("http") )
        {
            System.out.println( "Header is null or not http." );

            clientIn.close();
            clientOut.close();
            return false;
        }
        else
        {
            // Open socket to server

            URL url = new URL(header.url);

            int port = url.getPort();
            if(port < 1) port = 80;

            Socket sock;

            // Open a socket to either the host or to another proxy
            if ( proxyServer.getProxyFlag() )
            {
                sock = new Socket( proxyServer.getProxy(), proxyServer.getProxyNum() );
            }
            else
            {
                sock = new Socket(InetAddress.g​etByName(url.getHost​()),port);
            }
            
            serverIn = sock.getInputStream();
            serverOut = sock.getOutputStream();
            return true;
        }
    }

    /**
     * a request header is of this format:
     * Request-Line = Method SP Request-URI SP HTTP-Version CRLF
     *
     * this method assumes the second space delimited token is the request uri
     * and so it takes that token and strips the full url.
     *
     * this is done becuase with HTTP 1.0 the request header should contain
     * full URL information only if it is being sent to a proxy server, and
     * the proxy server is supposed to strip that information.
     */
    private String stripProxyInfoFromRe​questHeader() {

        //rewrite the uri
        String res = "";

        try
        {
            String origUrl = header.url;

            URL url = new URL(origUrl);
            header.url = url.getFile();
            
            res=header.toString();

            header.url=origUrl;
        }
        catch(MalformedURLException ex)
        {
            ex.printStackTrace();
        }
        
        return res;
    }

    /**
     * does the following <ol>
     * <li> get header info from client.
     * <li> strips all but relative address info from header
     * <li> sends that information to the rewritten header, along with all
     * the rest of the information to the server.
     *
     * <li> If the ProxyServer uses a proxy then, add the additional proxy header
     * info to the request and open the socket to the proxy.
     *
     * <li> gets the server response
     * <li> passes that information back to the client.
     * <li> the server should close its connection. once that happens, the
     * proxy closes its connection.
     * </ol>
     */
    public void run() {
        try {

            final int BUF_SIZE;
            {
                int rs = clientSocket.getRece​iveBufferSize();
                int ss = clientSocket.getSend​BufferSize();
                BUF_SIZE = rs<ss?ss:rs;
            }
     
            byte buf[] = new byte[BUF_SIZE];
         
            if( initClientServerConnections( clientSocket ) )
            {
                //read the header and strip proxy info
                String headerStr = stripProxyInfoFromRe​questHeader();
                
                //send the byte information to the server
                {
                    byte bytes[] = headerStr.getBytes();
                    serverOut.write(byte​s,0,bytes.length);
                }

                
                System.out.println( "");
                System.out.println( "-------------------");
                System.out.println( header );
                


                //next get the bytes for the rest of the request
                //and send them
                byte requestBody[];
                if(header.contentLength > 0)
                {
                    buffer.reset();
                    moveAvailableBytes(c​lientIn,serverOut,bu​f,
                        header.contentLength);
                    requestBody = buffer.toByteArray();
                }
                else
                {
                    requestBody = new byte[0];
                }
  
                //get the server reply
                buffer.reset();
                try {
                    moveAvailableBytes(s​erverIn,clientOut,bu​f);
                    proxyServer.processR​equest(header, requestBody,
                        buffer.toByteArray());
                }
                catch(IOException e) {
                    e.printStackTrace();
                }
  
                //the server should close its socket
                //automatically
  
                //close the client connection
                clientSocket.close();
            }
        }
        catch(IOException ex){
            ex.printStackTrace();
        }
    }

    private void moveAvailableBytes(InputStream src, OutputStream dest, byte buf[] )
    throws IOException
    {
        moveAvailableBytes(src, dest, buf, -1);
    }

    private void moveAvailableBytes( InputStream src, OutputStream dest, byte buf[], int max)
        throws IOException {

     int len;
     int num = 0;
            
     while ( ( max == -1 || num < max ) && 0 < ( len=src.read( buf, 0, buf.length ) ) )
            {
                dest.write(buf,0,len);
                buffer.write(buf,0,len);
                num += len;
     }
    }

}
Attachments

« Previous message in topic | 1 of 1 | Next message in topic »

Messages

Show all messages in topic

[maxq-users] Patch: Adding Proxy support to MaxQ fcohen Frank Cohen 2003-09-23 11:05:12 PDT
Messages per page: