Can't convert - My board's too large

[Fizzy]

How frustrating.

I have a Y1G SP1.1 board with a mere 17,000 posts over 1400 topics and with only 280 members.

I have installed SE 1.5.1 RC1 and it looks like just the ticket but can I convert my SP1.1 to RC1 ?
Not a chance.

I have been told by my host that they have a limit on the file size that can be imported into mySQL of a poxy 4 Megs in any 15 minute period. The messages in txt for the SP1.1 board total over 6 Megs.

When I run the converter supplied kindly by Compuart it bottoms out because of this stupid limit set by my host.
The best they can recommend is to import the messages in blocks to break it up into smaller chuckns, but I think the board needs to convert in one hit to maintain its relationships and integrity.

I have been pondering about getting round this problem and thought about being able to start the conversion, let it run for 3 megs the pause it for 15 minutes then run again the pause until the board has managed to upload itself
or
possibly find a way of importing the message files for separate sections/categories of the SP1.1 a cat at a time without corrupting the data and still maintaining integrity.

Is this a problem anyone else has come across ?
As this is my first ever conversion I am waaaaay out of my depth and would really appreciate any help or tips from anyone who can offer suggestions.

Thanks all

Fizz

[Compuart]

In order to reduce the size of the inserted data in the last step, you could skip all the log tables, as the're not necesary for the data integrity of the forum. This would mean:
delete (converter.php):

Code Select Expand

insertIt("{$dbprefix}log_mark_read", "(ID_MEMBER, ID_BOARD, logTime)", $insertData, 500);
and

Code Select Expand

insertIt("{$dbprefix}log_boards", "(ID_MEMBER, ID_BOARD, logTime)", $insertData, 500);
and

Code Select Expand

insertIt("{$dbprefix}log_topics", "(ID_MEMBER, ID_TOPIC, logTime)", $insertData, 500);

Be sure that you wait 15 minutes before doing the step that inserts the messages, and it might just be enough to get all messages converted.

[Fizzy]

Thanks so much for that tip Comuart.

I'll be sure to try it and let you know how I get on.
Ig it fails then I think the only alternative which I though of this morning would be to install onto my hard drive a mirror of the hosted RC1.
Then to run the converter from my PC, generate the mysql tables and relationships etc and then DUMP those new tables straight back on to the server in managable chunks.

Well wish me luck with Compuarts tweak - I have much more faith in "Big C's" ability than I do my own here

[Compuart]

Well wish me luck with Compuarts tweak - I have much more faith in "Big C's" ability than I do my own here

well, here goes nothing....

Good luck!

If that won't work I still have a more drastic option for ya, but for now I just assume this'll work

[Fizzy]

Well I really appreciate the help and the suggestion Big C, but unfortunately I have a sneaking suspicion that we were just slightly over the top with memory usage.

I even went hunting and tried a stab in the dark by inserting a

Code Select Expand

sleep(1000);
command in after "insert log topics" and just before "inserting messages" but it was a blind stab.

I guess the only other option (gulp !! ) is to reduce the board size by stripping out some older threads to bring it back down to less than 4 megs  

That's not my fave one though.

Big C, if your suggestion is for anything that doesn't involve branding irons, eating large spiders or immersing myself in a bath filled with excretions from a sceptic ulcer then i'm all ears. I really want to give my members YaBB SE as I know they'd love it.

Thanks again Big C - you've been a great help  

[Compuart]

Big C, if your suggestion is for anything that doesn't involve branding irons, eating large spiders or immersing myself in a bath filled with excretions from a sceptic ulcer then i'm all ears.

lol, I'll try to keep it as easy as possible

I sincerely hope this will work. I'll try to modify the converter so that step 5 is split in two parts. Step 5 will now add all category/board/topic/log-entry-info and 5b will insert all messages.

Good luck

(untested, but it should work... )

replace (convert.php):

Code Select Expand

<form action="converter.php?step=6" method="POST">
by:

Code Select Expand

<form action="converter.php?step=5b" method="POST">

replace (convert.php):

Code Select Expand

         print "</blockquote>";

         // Getting message info
by:

Code Select Expand

         print "</blockquote>";
      }
   }
?>
</td></tr></table>
<center>
<input type=hidden name="y1gdir" value="<?=$y1gdir?>">
<input type=hidden name="boardsdir" value="<?=$boardsdir?>">
<input type=hidden name="membersdir" value="<?=$membersdir?>">
<input type=hidden name="datadir" value="<?=$datadir?>">
<input type=hidden name="vardir" value="<?=$vardir?>">
<input type=hidden name="dbuser" value="<?=$dbuser?>">
<input type=hidden name="dbpassword" value="<?=$dbpassword?>">
<input type=hidden name="dbprefix" value="<?=$dbprefix?>">
<input type=hidden name="dbname" value="<?=$dbname?>">
<input type=hidden name="dbserver" value="<?=$dbserver?>">
<input type=hidden name="installdir" value="<?=$installdir?>">
<input type=submit value="Proceed >"> (Only proceed if successful!)
</center>
</form>
<?php
}
elseif($step=="5b")
{
?>
We are now going to attempt to convert your message files.  This may take a while.<br />
<form action="converter.php?step=6" method="POST">
<table border=0 cellspacing=0 cellpadding=3>
<tr><td>
<?php
   if (1)
   {
      if (1)
      {
         //retrieving topic ID's
         if (count($insertData) > 0)
         {
            $request = mysql_query("SELECT TEMP_ID_TOPIC, ID_TOPIC FROM {$dbprefix}topics");
            while ($row = mysql_fetch_array($request))
               if ($row['TEMP_ID_TOPIC'] > 0)
                  $topics[$row['TEMP_ID_TOPIC']] = $row['ID_TOPIC'];
         }

         // get all users from the database that were successfully imported and that also exist in Y1G
         print "Initializing member data for processing<br />";
         $users = array();
         $request = mysql_query("SELECT memberName, ID_MEMBER, emailAddress FROM {$dbprefix}members");
         while ($row = mysql_fetch_array($request))
            if (file_exists($membersdir . '/' . $row['memberName'] . '.dat'))
               $users[$row['memberName']] = $row['ID_MEMBER'];

         // Getting message info


[Fizzy]

Cheers for that Big C

I followed the instructions and we definitely made better progress that time.

Id did howver throw up this error

We are now going to attempt to convert your message files. This may take a while.

Initializing member data for processing

Warning: Access denied for user: 'dconlon@localhost' (Using password: NO) in /home/s/k/skincell/public_html/yabbse/converter.php on line 808

Warning: MySQL Connection Failed: Access denied for user: 'dconlon@localhost' (Using password: NO) in /home/s/k/skincell/public_html/yabbse/converter.php on line 808

Warning: MySQL: A link to the server could not be established in /home/s/k/skincell/public_html/yabbse/converter.php on line 808

Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource in /home/s/k/skincell/public_html/yabbse/converter.php on line 809
Importing Messages

Warning: Invalid argument supplied for foreach() in /home/s/k/skincell/public_html/yabbse/converter.php on line 817

I don't know if that's easy to fix or not ? ?
But as a thought, is it possible to write in to the code for a 1 second delay after each message is converted? Just a thought that it might buy the time needed to stay under the 4 Meg ceiling ?

I really appreciate you taking the time to try and help me with this Big C. It's great to see that there is this level of support for us mere mortals.

[Compuart]

oops, I forgot to connect to the database  

replace:

Code Select Expand

<?php
   if (1)
by:

Code Select Expand

<?php
   $dbcon = mysql_connect($dbserver, $dbuser, $dbpassword);
   mysql_select_db($dbname);
   if (1)

hope this works

[Fizzy]

Hey there Big C,

Well you seem to be making real progress
No error messages saying that the memory went on vacation but I did however get the following

We are now going to attempt to convert your message files. This may take a while.

Initializing member data for processing
Importing Messages

Warning: Invalid argument supplied for foreach() in /home/s/k/skincell/public_html/yabbse/converter.php on line 821

which meant that the messages didn't import.

I'm not sure what would be the best thing to do now, what do you think? I get the feeling that we are that close to solving this I can almost taste it.

[Compuart]

ok, we're slowly getting there, but we will get there

replace:

Code Select Expand

        if (count($insertData) > 0)
        {
           $request = mysql_query("SELECT TEMP_ID_TOPIC, ID_TOPIC FROM {$dbprefix}topics");
           while ($row = mysql_fetch_array($request))
              if ($row['TEMP_ID_TOPIC'] > 0)
                 $topics[$row['TEMP_ID_TOPIC']] = $row['ID_TOPIC'];
        }
by:

Code Select Expand


           $request = mysql_query("SELECT TEMP_ID_TOPIC, ID_TOPIC FROM {$dbprefix}topics");
           while ($row = mysql_fetch_array($request))
              if ($row['TEMP_ID_TOPIC'] > 0)
                 $topics[$row['TEMP_ID_TOPIC']] = $row['ID_TOPIC'];

* Compuart hopes this will do it...

[Fizzy]

Hey there Big C

It's nearly there.

We are now going to attempt to convert your message files. This may take a while.

Initializing member data for processing
Importing Messages

Warning: Invalid argument supplied for foreach() in /home/s/k/skincell/public_html/yabbse/converter.php on line 818

foreach($topics as $OLD_ID_TOPIC => $ID_TOPIC)
{
if (checkFile("$datadir/$OLD_ID_TOPIC.txt"))
{
$messageFile = file("$datadir/$OLD_ID_TOPIC.txt");
foreach($messageFile as $messageFileEntry)
{
if (trim($messageFileEntry != ''))

Line 818 at the top.

I noticed that it is now running the conversion much faster so you must nearly be there Big C

[Fizzy]

Well here's how we left it, my server host has a mere 4 meg mysql transfer limit every 15 minutes.

Big C has literally tweaked the converter to the limit to try and accommodate my poxy server hosts.

We finally called it a day when we realised that the converter would not be able to do the impossible, namely to fit a pint in a half pint glass.

My final solution however did the biz.

I ended up installing Apache, mysql and PHP support on my home PC with the help of Firepages PHPDev. I installed YaBBSE, ran the converter linking in to a backup of the SP1.1 forum om my hard drive and then split down the import procedure onto my host server in managable chunks.

THE CONVERTOR FROM COMPUART WORKED A TREAT FIRST TIME !

I just wanted to log the solution in case any other guys get this problem and most importantly to publicly thank the BIG C for bending over backwards trying to help me and for providing such a terrific convertor in the first place

Big C - You da man !

[Compuart]

I'm glad you've solved the problem and everything works now

[GWP]

Compuart
I am just in the process of moving my site to a different server so i can use SE as opposed to SP1.1 (which i use at the moment)
I am not sure if this is the place to ask, but will give it a try...

" Will i be able to convert all my members posts and IM's etc into my new board.?"
This is very important to me and i don't want to go through the entire process only to find that I can't.

Finally, I have downloaded the latest fully working version of YabbSE (from: http://cbeta.yabbse.org/yabbse_1-5-1RC1B42.zip ), will the new version (1.5.1RC1) be an upgrade option, or will I have to start again.

Thanks

[[Unknown]]

Every question you asked has the same answer:

Yes.

-[Unknown]