Multithreaded PHP

Rob · January 3rd, 2007, 11:38 PM

Whilst working on a new webmaster tool (which I will unleash to you lot in the future) I ran into an issue...

The tool performs dozens of checks on a web page and then returns a report.

The problem was that the speed was very very slow indeed and I needed a way to speed it up.

What I needed was a way to handle these checks in a multithreaded way (or at least a pseudo multithreaded way) so I rang up the only person I knew who could offer a suggestion (Graham Ellis of Well House Consultants)

He suggested a workaround and whilst it isnt true multithreading, it added severe speed to my application.

Here are the two core files which make it all happen.

This first file is the main php file you hit in your browser and for the sake of this example, needs a url passed in the querystring. The url is then passed to the 6 scripts by way of a command line parameter (yes, you read that right)

Code:

<?php
 
$url = $_GET['url'];
$scripts = array(
   "script1.php",
    "script2.php",
    "script3.php",
    "script4.php",
    "script5.php",
    "script6.php");
 
# Set off parallel contacts to each of the scripts
 
for ($p=0; $p<count($scripts); $p++) {
    $fh[$p] = popen("/usr/local/bin/php ".
    "/home/yoursite/public_html/" . $scripts[$p] ." " .
    $url,"r");
    }
 
########################################
# DO AS MUCH PROCESSING IN HERE AS YOU LIKE
########################################
# your scripts here
# code here is processed whilst the above script calls are being processed
#
#
#
#
 
# We're done with our work - read back the results from the scripts
 
for ($p=0; $p<count($scripts); $p++) {
    $st = fgets($fh[$p]);
    $rs .=  $scripts[$p] . " : " . $st ."<br />";
    }
 
print $rs;
?>

What the above does is fire off the 6 scripts in parallel, passing $url to each script as a command line parameter.
It executes each script by executing the php binary itself, so the processes themselves live outside of apache.
The instant the scripts are fired, they work in the background whilst control is passed back to php to continue.
After we've done other intensive processing (whatever code you insert above) we can finally get all the scripts return values.

For this example, here is the code for one of the scripts that is called - note how it reads the url we passed from the command line

Code:

<?
$url = $argv[1];
// code goes here
// in my webmaster tool, each of these scripts does a function
// such as check pageRank, alexa rank, backlinks, etc.
print "this is the returned value";
 ?>

The above method is best used on very intensive tasks that typically take a long time.

From my own tests, it's best to split up around 50% of the work. With that 50%, farm it out to a few different scripts and execute them with the above method. The other 50% can be processing within the main script before you finally read the results back in.

There are probabally better ways to further optimise and maybe better ways to split up the work load, but all of that experimentation I shall leave up to you.

Obviously this method can only be used in certain situations like mine... it wont make any differnce to scripts that are processor intensive, but for those which are not - this will reall add some speed if your script is suitable for splitting up in this way.

Be sure to post here with any info you gather.

Happy Coding

Digg this article

grahame · January 4th, 2007, 05:04 AM

Many thanks for the "plug", Rob ... the work you've done and shown me is fascinating - I'll be coming back and running it against a sampling of my pages and it should be a great help to show us how we can improve our placement.

I have used similar techniques (though through Tcl/Expect or Python) in the past to check which systems are available on a network. It was a similar case to yours in that I wanted to check up on some 20 hosts, but couldn't afford to wait for serial ping timeouts. So I "paralleled" them. To this day, I have a Tcl page on our main web sites that, in a single click, tells me about some 10 different servers on which we look after sites ... and I can check from anywhere in the world.

One caution for anyone using this technique. I believe that pipes have a 4k buffer in them, so your popen'd scripts could come to a temporary stop if they generate a lot of output and haven't been "reaped".

Ryan Fait · January 4th, 2007, 06:39 AM

That's cool! I can seriously use that in a project I'm working on!

Tim356 · January 4th, 2007, 06:50 AM

If it works as you say it does, then I will also be able to use it in a current project. Thanks guys!

ropu · January 31st, 2007, 03:00 PM

hi, i also needed some multithreading/parallel task.
i had a different approach, using WebService like interface and sockets.

I have the main file, socket_cli_test.php, the one called by the user in the Browser.

That files calls internally n other ones via socket connection (HTTP request) and then grabs the answers. Ive also implemented time out, because i needed the whole process to finish before 3 seconds.

here is the code:
socket_cli_test.php

Code:


			
<?php
/*
 * Created on 31/01/2007
 *
 * Coded by: Ropu
 *  - Buenos Aires, Argentina  - z-tests
 */
error_reporting(E_ALL);
$cant_sockets = 9;
$timeout = 2.9;

// Write
$sockets = array();
$URI = '/~ropu/tests/socket_srv_test.php';
$Host =  'localhost';
$port = 80;

echo "start\n<xmp>";

list($start_m, $start_s) = explode(' ', microtime());
$start = $start_m + $start_s;

// Create a new socket
for($i=0; $i<$cant_sockets; $i++) {
    $sockets[$i] = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
}

// Connect to destination address
for( $i = 0; $i < $cant_sockets; $i++ ) {
    socket_connect($sockets[$i], $Host, $port);
//    socket_set_nonblock($sockets[$i]);
    socket_set_option($sockets[$i], SOL_SOCKET, SO_RCVTIMEO, array("sec" => 0, "usec" => 500000));
}

for($i=0; $i<$cant_sockets; $i++) {
    $ReqBody = $i;
    $ReqHeader =     "POST $URI HTTP/1.1\n".
                            "Host: $Host\n".
                            "User-Agent: Ropu Soft\n".
//                            "Authorization: Basic ".base64_encode($merid.':'.$merkey) . "\n".
                            "Content-Type: application/xml\n".
                            "Connection: close\n". // Important HEADER!!! to avoid delays when doing do{}while(reading)! 
                            "Content-Length:  ". strlen($ReqBody) ."\n\n". // without this one, post wont work
                            "$ReqBody\n";
    socket_write($sockets[$i], $ReqHeader);
}

$mid = $start;
while($mid-$start < $timeout){
    $new_socks = $sockets;
    // Calculate the remaing time before timeOut
    $working = $timeout - ($mid-$start);
    $rsec = (int)($working);
    $rusec = (int)(($working - $rsec) * 1000000);
    
    // Watch for activity in any socket
    $num_changed_sockets = socket_select($new_socks, $write = NULL, $except = NULL, $rsec , $rusec);
    for($i=0; $i<count($new_socks); $i++) {
        // Read all info in the socket
        do {
            echo $read = socket_read($new_socks[$i], 2048);
        }while(!empty($read));
//        echo "sock: ". $i . " => " .socket_strerror(socket_last_error($new_socks[$i])) ."\n";
    }
    // Discard used sockets
    $sockets = array_diff($sockets, $new_socks);
    // If all sockets are read, don't wait for the timeOut.
    if(count($sockets) < 1)
        break;
    // Calculate actual time
    list($mid_m, $mid_s) = explode(' ', microtime());
    $mid = $mid_m + $mid_s;
}

list($end_m, $end_s) = explode(' ', microtime());
$end = $end_m + $end_s;
echo "\n\nTardó: ". ($end-$start) ." segs"; 

?>

socket_srv_test.php

Code:


			
<?php
/*
 * Created on 31/01/2007
 *
 * Coded by: Ropu
 *  - Buenos Aires, Argentina  - z-tests
 */

// set_time_limit(1);
//print_r($_POST);
//print_r($HTTP_RAW_POST_DATA); echo "\n";
$_POST['id'] = $HTTP_RAW_POST_DATA;
 $id = rand(100, 100000);
 $message_log = fopen('test.log', "a");
 fwrite($message_log, sprintf("\n\rIn: %s:- Id: %d %d\n",date("D M j G:i:s T Y"), $id, $_POST['id'] ));
 echo "In: ".date("D M j G:i:s T Y") . " Id: $id " . $_POST['id'] ."\n";
 
 $delay = rand(1,5);
// if($_POST['id'] == 0)
//     $delay = $_POST['id'] + 2;
//$delay = 2;
 sleep($delay);

 fwrite($message_log, sprintf("\n\rOut: %s:- Id: %d %d Delay: %d\n",date("D M j G:i:s T Y"), $id,  $_POST['id'], $delay));
 echo "\nOut: ".date("D M j G:i:s T Y") ." Id: $id " . $_POST['id'] ." Delay: $delay\n\n";
 echo "--------------------------------------------------\n\n";
 //print_r($_SERVER);
 
?>

Feel free to send some feedback and suggestions.

hope it helps,

ropu

Ryan Fait · January 31st, 2007, 06:38 PM

Thanks for sharing!

matseth · May 4th, 2007, 08:08 PM

Hello
can anyone help me??
I am doing the code that Rob giving but there no output it only print out the name of the file.

Is this code supported in window xp because i want to do php multithread.

What i know that php multithread and forking is not supported in window platform and if supported it can only be used in CLI am i right.

plzz someone help me i am stuck to do my project

Accurax · May 4th, 2007, 08:42 PM

Grahame... i think its true to say we all love you ... :wub: :wub:

jasegura · December 20th, 2008, 10:29 PM

Hey guys, hi, you know, I think you must take a look at this:
http://php.net/manual/en/ref.pcntl.php
What is this?. Well, you are talking about threads, I don't know if you know what is the difference between threads and a processes, but an option to make things go "concurrent" or "parallel" ( it depends if you are working on a multiprocessor system, or in a monoprocessor system, anyway you could make things running faster and more efficiently ) is to use multiprocessing way. You cannot create (not yet) threads in PHP, but you can create child processes (or cuncurrent or parallel processes). If your server is running on Linux family OS, this is very importan, "This PHP extension is not available on Windows platforms", you can use the PCNTL PHP extension, this is a "Process Control support in PHP implements the Unix style of process creation, program execution, signal handling and process termination.".
For example:

Code:


			
<?php

$pid = pcntl_fork();         // child process creation
if ($pid == -1) {              // pcntl_fork() returns -1 if something went wrong
     die('could not fork');
} else if ($pid) {              // else, for the child process $pid is zero, but for the
                                       // parent process is the Process ID or PID of the
                                       // new child process.
     // here we are the parent section
     pcntl_wait($status);    // waiting for the child process to finish (this is a very good
                                       // practice, if you don't do this, you could make the system
                                       // crash, because if you just let processes like Orphans or 
                                       // Zombies no more PIDs will be available for new 
                                       // processes, so you have to wait for each process to 
                                       // finish before exit from the script).
} else {
     // we are the child
     echo "I'm the child process ;-). Wait for me, don't let me be an Orphan or a Zombie!!!";
}

?>

This sample, is the solution to your parallelism problem, you can easy picture this like the following:

<?php
// some code, runing on then main process
// you make "fork" or "pcntl_fork" in this case
// so, you have two processes (the main process -father-, and the new process -son-) running parallel or concurrent to each other, they "fight" to use the processor/s.
// never forget to wait for all the child processes you have created!!
// and exit
?>

The new process, the child one, is an exact copy of the main process, with its own memory slice, it is like to do Ctrl + C and Ctrl + V with the processes on main memory, the only difference between them is, the pcntl_fork returns zero for new child process, and an integer value to the parent process, that is the process id of the new process on your system. That's the way you use to know if you are running on child process or parent process.

If you read more about this, you will be able to do some interesting stuff, like "client-server" application, using sockets, or faster link checker (one process for each url to check, running parallel to each other), and much much more interesting stuff.

Just study this: http://www.php.net/manual/en/intro.pcntl.php

Bye,

Happy coding for everyone! =)

Javier.