• Typo3MnogoSearch

Typo3 MnogoSearch Engine Setup for Ubuntu 09.10 and 10.04

Base for the setup is the manual.swx file from Dmitry and http://software.rde.de/mnogo-die-erste.html . I will just describe the adoptions for Ubuntu 09.10 and 10.04.

Installation Requirements

The following packages are required for compiling mnogosearch.

automake - installed
autoconf - installed
gcc - installed
ispell - to install
ingerman - to install
libmysqlclient16-dev - to install
mc - for editing, installed
mysql - installed
php5-dev - to install
zlib - installed

Now the source code is copied to /opt and unpacked.

$ sudo cp mnogosearch-3.3.9.tar.gz /opt/mnogosearch-3.3.9.tar.gz
$ cd /opt
$ sudo tar xzf mnogosearch-3.3.9.tar.gz
$ cd mnogosearch-3.3.9

Next step is to modify, compile and install mnogosearch.

Edit files to exclude PGSQL (manual.swx, page 13)
Edit file "configure.in"
Line 1017,  AC_DEFINE([HAVE_PGSQL], [1], [Define if you want to use PostgreSQL])
change "1" to "0"
Edit file "configure" to exclude PGSQL
Line 26547, #define HAVE_PGSQL 1  # 1 -> 0
$ ./configure ... # manual page 13
Now open include/udm_autoconf.h and locate /* #undef HAVE_PGSQL */
Line 133, change to "#undef HAVE_PGSQL"
$ make
$ sudo make install

Next step is to modify, compile and install the PHP extension. In order to check the proper installation look in the TYPO3 Install tool under 8: phpinfo(). Under chapter Configuration it should show the block mnogosearch with Support enabled.

cd /opt/mnogosearch-3.3.6/php
Edit php_mnogo.c - AFTER line 30 #include "php.h"
Insert #undef HAVE_PGSQL
$ phpize
  Configuring for:
  PHP Api Version:         20041225
  Zend Module Api No:      20060613
  Zend Extension Api No:   220060519

$ sudo ./configure --with-mnogosearch=/opt/mnogosearch 
$ sudo make
$ sudo make test
$ sudo sudo make install
  Installing shared extensions:     /usr/lib/php5/20060613+lfs/
OK
Just for special case: Find PHP5 extension dir:
$ php -i | grep extension_dir
$ sudo cp modules/mnogosearch.so /usr/lib/php5/20060613+lfs/mnogosearch.so

Edit php.ini for Command Line Interpreter testing.

/etc/php5/apache2/php.ini
and
/etc/php5/cli/php.ini
Line 503 extension_dir = "/usr/lib/php5/20060613+lfs"
At the end of "Dynamic Extensions" add:
Line 616 extension=mnogosearch.so

$ sudo /etc/init.d/apache2 restart
$ php -m   # show all loaded modules
First create the user '''_cli_mnogosearch''' in Typo3 backend.
$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch help  # for test

Create database index.

$ mysql -u root -p rudisflugis
Enter password: ****
> create database mnogosearch;
> grant all privileges on mnogosearch.* to 'root'@'localhost' identified by 'rudiscomplexpassword';
> exit;
$ find /usr -name "cli_*"
/var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -w -n &>/dev/null
or
$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -w -v 3  # verbose
or clear index
$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -x -Cw

TYPO3 setup

Now login the TYPO3 backend and create a new user _cli_mnogosearch. The password must be set, but is not used.

Admin-Tools/Administration Result usernames: - click on Add + symbol
Enter usename "_cli_mnogosearch"
Enter password "password"
Save record

In order to allow global extensions, it must be enabled once with the Install-Tool.

Start Install-Tool
Select 5:All Configuration
Set a mark at [allowGlobalInstall]
At the bottom of the windows click on button "Write to localconf.php"

Next start the extension manager.

Mode: Import extensions
Download first from www.typo3.org the extensions as *.t3x file:
  page path, ver. 0.1.6 Dmitry Dulepov
  universal page browser,  ver. 1.1.0 Dmitry Dulepov
  mnogosearch, ver. 2.2.2 Dmitry Dulepov
Select function "Import extensions"
in field "UPLOAD EXTENSION FILE DIRECTLY (.T3X)" select
  pagepath_0.1.6.t3x
... to location "Global (typo3/ext/)"
mark "Overwrite any existing extension!"
click on "Upload extension file"
No file uploaded! Probably the file was too large for PHPs internal limit for uploadable files.
------------- unfortunately it does not work, also with ver. 4.2.12 -----------
Solution: $ sudo mkdir /var/run/typo3-dummy
          $ sudo chown www-data:www-data /var/run/typo3-dummy

Workaround:
First install "local" from repository, than move to global folder.

Type in the "Look up" field:
universal page browser: Look up for "page"
mark "Overwrite any existing extension!"
click on "Upload extension file"
Repeat for Look up "pagepath"
Repeat for Look up "mnogosearch"
Then move the 3 folders from /var/lib/typo3conf/ext to /usr/share/typo3/typo3_src-4.2.12/typo3/ext
$ sudo chown -R www-data:www-data /usr/share/typo3/typo3_src-4.2.12/ext
$ cd /var/lib/typo3-dummy
$ sudo  chmod -R a+w fileadmin typo3conf typo3temp uploads

When the last extensions mnogosearch was installed, the extension manager tells Installing mnoGoSearch: DATABASE NEEDS TO BE UPDATED and Before the extension can be installed the database needs to be updated with new tables or fields. Please select which operations to perform:

Add tables
  mark at CREATE TABLE tx_mnogosearch_indexconfig 
  mark at CREATE TABLE tx_mnogosearch_urllog 
Create folder
  mark at Create directory "typo3temp/mnogosearch/var"
Click on "Make updates"

You will see a new windows with:
Active status:
The extension is installed (loaded and running)!
then check:
  Path to mnoGoSearch installation: /opt/mnogosearch
  Database connection string: mysql://user:pass@host/dbname/?dbmode=multi
    mysql://root:o****@localhost/mnogosearch/?dbmode=multi  # ReadMe_Typo3.txt
  Charset for database: utf-8
  Charset for results: utf-8
  mark at Hide internal tables in List module
then click on "Update"

then check:
  Installation status:
    Type of install: Global - Global extensions (typo3/ext/) - Available for shared source on server (Dynamic).

On the command line test the function:
NOTE: options -h, --help and -? does NOT work, use instead just "help"!

rudi@rudi2u:~$ /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch help
Usage: cli_mnogosearch.phpsh [-n] [-w]

This script reindexes TYPO3 web sites as defined by current TYPO3 configuration.
It accepts the following options:

-c                Only check and create database if necessary. Do not reindex.
-d                Display generated indexer configuration and exit.
-n                Force reindexing of new URLs (normally should be set)
-p pid            Process indexing configuration only from this pid
-w                Create statistic for misspelled words. Useful only if
.                   Ispell dictionaries are included to mnoGoSearch
.                   configuration (see mnoGoSearch documentation)
--dry-run         Show what will be done (not applicable to -d and -E)
-h, --help, -?    Display this help message
-x                Pass the argument to mnoGoSearch indexer
-v level          Be verbose. Level is 0-5. Default is 0 (complete silence)

Update to TYPO3 4.2.12

Because the included (Ubuntu 0910) TYPO3 version was 4.2.6 it should be updated:

Download from www.typo3.org file typo3_src-4.2.12.tar.gz
Unpack with archive manager in folder /usr/share/typo3

Copy global extensions from old old installation:
$ sudo cp -R /usr/share/typo3/typo3_src-4.2/ext/* /usr/share/typo3/typo3_src-4.2.12/ext
$ sudo chown -R www-data:www-data /usr/share/typo3/typo3_src-4.2.12/ext
$ cd /var/lib/typo3-dummy/
$ sudo rm typo3_src
$ sudo ln -s /usr/share/typo3/typo3_src-4.2.12 typo3_src
$ cd /usr/share/typo3/typo3_src-4.2.12
$ sudo chown -R root:www-data *
$ sudo chmod -R g+w *

Calling the Install-Tool from Backend gives several warnings, but it works.

TYPO3, mNogoSearch setup

In the TYPO3 backend click on Web/List and add a new page named Konfigurations-Tools.

Typo3 4.2.x
Click on globe and "new", new page with wizzard,
   page title is "Konfigurations-Tools" - save and exit
Click on right side on "Create new record", select type "mnoGoSearch indexing configuration" 
  and insert it below the "Master_Template".
The title will be "mnoGoSearch-IndexConfig"
Indexing path: http://www.rudiswiki.de/flugis/  # rudisflugis, IMPORTANT: trailing "/"
   otherwise other web pages from this domain (e.g. wiki1) will be indexed.
Typo3 4.3.x
Select Web/List mode.
  Click on the domain root in the tree.
  In the right window click on the "New record" icon.
    Select "new page" and name the title "Konfigurations-Tools"
    Click in the page tree on "Konfigurations-Tools", then in
      the right window on the icon "Create new record".
      Next click on "mnoGoSearch indexing configuration"
        Configuration type: server
        Indexing path: http://www.rudiswiki.de/flugis
        Title: mnoGoSearch root indexer
        Index method: Allow
        Index period:  (use cron job)
        Save the record.

In case you changed the indexing path, please lookup with phpmyadmin if you have just the desired inding pathes in the table xxx. If there are more than that, delete the unwanted indexing pathes.

First Index run:

clear index
$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -x -Cw
or index
$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -v 3 -w
indexer from mnogosearch-3.3.9-mysql-DB2-solid-SAPDB-ibase-ctlib-freetds-oracle8-oracle8i started with '/tmp/mnogosearch-olrMiC'
[8230]{01} URL: http://localhost/typo3
[8230]{01} ROBOTS: http://localhost/robots.txt
[8230]{01} URL: http://localhost/typo3/
[8230]{01} URL: http://localhost/typo3/link-seite.html
...
[8230]{01} URL: http://localhost/typo3/fileadmin/download/impressum0.html
[8230]{01} Writing words (177107 words, 3978987 bytes, final).
[8230]{01} The words are written successfully. (final)
[8230]{01} Done (358 seconds, 562 documents, 8242536 bytes, 22.48 Kbytes/sec.)
Calculating word statistics
Word statistics done

Add Search Mask to web

In order to enter search words, it needs a search mask. This is imported from a master web, schnelle_suche.t3d CSS

Exclude anything else content from index

In order to exclude anything else content from the index, use the mark up <NOINDEX> .. </NOINDEX> in the proper template xxx.html file, e.g. fileadmin/templates/rudi-folgeseite-index.html.

mNogoSearch PageBrowser

In order to use the page browser to navigate in more than one page of search results, it has to be setup first.

Edit file
/var/lib/typo3-dummy/typo3/ext/mnogosearch/static/mnoGoSearch/constants.txt 
Add on top of the file from /var/lib/typo3-dummy/typo3_src/typo3/ext/pagebrowse/constants.txt:
plugin.tx_pagebrowse_pi1 {
        # cat=plugin.tx_pagebrowse_pi1; type=string; label=Template file
        templateFile = EXT:pagebrowse/res/template.html
}

Edit file
/var/lib/typo3-dummy/typo3/ext/mnogosearch/static/mnoGoSearch/setup.txt 
Add on top of the file from /var/lib/typo3-dummy/typo3_src/typo3/ext/pagebrowse/setup.txt:
plugin.tx_pagebrowse_pi1 = USER
plugin.tx_pagebrowse_pi1 {
        # USER_INT support
        includeLibs = EXT:pagebrowse/pi1/class.tx_pagebrowse_pi1.php
        userFunc = tx_pagebrowse_pi1->main

        # Template file
        templateFile = {$plugin.tx_pagebrowse_pi1.templateFile}

        # Extra parameters to the query string. Must start with & if not empty
        extraQueryString =

        # Number of page links to show before the current page
        pagesBefore = 3

        # Number of page links to show before the current page
        pagesAfter = 3

        # GET variable name for page pointer. Examples: "page" or "tx_exykey_pi1|page". Notice that array separator is pipe (one level only!)
        pageParamName = tx_pagebrowse_pi1|page

        # Enables section for "more" pages. This section is shown after links to next pages, normally like three dots (1 2 3 ...). Notice that you can also hide it by emptying corresponding template section.
        enableMorePages = 1

        # Enables section for "less" pages. This section is shown after links to next pages, normally like three dots (1 2 3 ...) Notice that you can also hide it by emptying corresponding template section.
        enableLessPages = 1
}

Edit /var/lib/typo3-dummy/typo3/ext/mnogosearch/resources/styles.css 
Add on top of the file from /var/lib/typo3-dummy/typo3_src/typo3/ext/pagebrowse/styles.css:
/* der Dmittry pagebrowser */
.tx-pagebrowse { text-align: center; list-style: none inside none; padding: 0 0; margin: 0px; ; border:1px solid orange; font-weight: bold; ; background-color: #FFFFCC}
.tx-pagebrowse li { display: inline; list-style: none; margin: 0 0; padding: 0 5px 0 0; }
li.tx-pagebrowse-pages { width: 80%; }
li.tx-pagebrowse-pages ol { display: inline; margin: 0 0; padding: 0 16px; }

Clear cache - It works

For the German translation the following files must be there (translated):

/var/lib/typo3-dummy/typo3conf/I10n/de/mnogosearch/de.locallang_db.xml
/var/lib/typo3-dummy/typo3conf/I10n/de/mnogosearch/pi1/de.locallang.xml
/var/lib/typo3-dummy/typo3conf/I10n/de/pagebrowse/pi1/de.locallang.xml

Take care that all folders and files have user:group www-data:www-data.

In order to have the pagebrowse navigation also on top of the results field add:

        <div class="tx-mnogosearch-page-browser">
                ###PAGE_BROWSER###
        </div>
before line:
        <div id="tx-mnogosearch-results">
to the file: /var/lib/typo3-dummy/typo3/ext/mnogosearch/resources/template.html
A reload in the web browser will show the change.

Highlight Search key words

In order to highlight the search key words more obvious, you have to edit the file ext/mnogosearch/resources/styles.css and replace .tx-mnogosearch-pi1-result-excerpt-selection to:

.tx-mnogosearch-pi1-result-excerpt-selection {
    font-weight:b; 
    color:red !important; 
    background-color:yellow; 
    padding:0px 2px 0px 2px; 
}
or just remove the comment:
    /* */
    border: 1px dotted silver;
    margin: 0 3px;
    padding: 2px 2px;
    /* */
    font-weight: bold;

Unfortunately when you click on the found page link, the search key words are not highlighted on this web page. In order to add that function you have to modify the PHP program ext/mnogosearch/model/class.tx_mnogosearch_model_results.php  either local in /var/lib/typo3-dummy/ or global in /var/lib/typo3-dummy/typo3/. At about line 112 you add the line:

    $result->url = $result->url."?&no_cache=1&sword_list[]=".$pObj->piVars['q']; 

That adds the necessary commands to the URL and the search key words, as you can see in the last line of the search result.

In order to figure out in which PHP program the search word list is processed, you can search for:

rudi@rudi2u:/var/lib/typo3-dummy/typo3$ grepall sword_list
./ext/mnogosearch/model/class.tx_mnogosearch_model_results.php
./sysext/cms/tslib/class.tslib_search.php
./sysext/cms/tslib/class.tslib_pagegen.php
./sysext/cms/tslib/class.tslib_fe.php
./sysext/indexed_search/pi/class.tx_indexedsearch.php

grepall utility, for searching text in a folder.

 #!/bin/sh
 # grepall utility, 2009-11-27 from Kofler Linux p.515
 find -type f -exec grep -q $1 {}  \; -print

If you click on the result link, the search word is highlighted and has a red border. In order to remove the red border search in file fileadmin/css/rudi-content.css for csc-sword and remove the text:

border:1px red solid;

depreciated error in TYPO3 version 4.3.x

If you click on a link of a search result you get an error message:

Function sql_regcase() is deprecated in /usr/share/typo3/typo3_src-4.3.3/typo3/sysext/cms/tslib/class.tslib_pagegen.php line 173 

Solution is in: http://www.typo3.net/forum/list/list_post//97671/ and http://developer-services.net/?p=18

Add a line in /etc/typo3-dummy/localconf.php
$TYPO3_CONF_VARS['SYS']['exceptionalErrors'] = 22527;

Add a line in /etc/php5/apache2/php.ini
date.timezone = Europe/Berlin

$ sudo /etc/init.d/apache2 restart

mnogosearch debug / info

In order to debug mnogosearch you can use the -d command line option to show all setup parameters:

rudi@rudi2u:~$ sudo /var/lib/typo3-dummy/typo3/cli_dispatch.phpsh mnogosearch -v 3 -d
# Defaults
DBAddr mysql://root:password@localhost/mnogosearch/?dbmode=multi
VarDir /var/lib/typo3-dummy/typo3temp/mnogosearch/var
PopRankUseShowCnt yes
PopRankUseTracking yes
PopRankFeedBack yes
PopRankSkipSameSite no
Section body 1 8092 html
Section title 2 256 text
Section meta.keywords 3 64 text
Section meta.description 4 64 text
SaveSectionSize yes
LocalCharset utf-8
BrowserCharset utf-8

#Allow *.htm *.html *.php *.txt */ *.pdf *.doc *.odt *.swx
Mime message/rfc822 text/plain "cat $1"
Mime application/msword text/plain "catdoc -a $1"
Mime application/pdf text/plain "pdftotext -enc UTF-8 $1 -"
Mime application/vnd.ms-excel text/html "xltohtml $1"
Mime application/vnd.ms-powerpoint text/html "pptohtml $1"
Mime application/vnd.oasis.opendocument.text text/plain "/usr/bin/php -q '/usr/share/typo3/typo3_src-4.2.12/typo3/ext/mnogosearch/cli/catoo.php' '$1'"

Disallow *.rdf *.xml *.rss *.js *.css *.jpg *.png *.gif *.svg *.swf *.tif *.tiff
Disallow *.bz2 *.gz *.zip *.xpi *.dmg *.exe *.scr *.avi *.jpeg *.ods *.psd *.rar
Disallow *.fla Thumbs.db *.rpm *.rm *.qt *.mov *.pfb *.ttf* *.fon *.mpg *.mpeg *.ai
Disallow *.t3x *.t3d *.dll *.dat *.cab *.inf

AddType application/vnd.oasis.opendocument.text *.sxw *.odt
AddType text/plain *.txt
AddType text/html *.html *.htm
AddType text/rtf *.rtf
AddType application/pdf *.pdf
AddType message/rfc822 *.eml
AddType application/msword *.doc *.dot
AddType application/vnd.ms-excel *.xls
AddType application/vnd.ms-powerpoint *.ppt

HoldBadHrefs 2d
DetectClones yes
HTTPHeader "X-TYPO3-mnogosearch: 2de8d32bea4c53eca8f5a3ca88951fbf"
DefaultContentType "text/html"
# uid=2
Period 10y
Server http://rudiswiki.homelinux.org/typo3/

Disallow special file types

In order to disallow special file types like *.wmv or *.bmp you have to add that to a list in file ext/mnogosearch/cli/cli_mnogosearch.php either local in /var/lib/typo3-dummy/ or global in /var/lib/typo3-dummy/typo3/. Another option is to exclude printview files, with *.98.html.

Line 387
Disallow *.rdf *.xml *.rss *.js *.css *.jpg *.png *.gif *.svg *.swf *.tif *.tiff
    Append *.bmp   # to exclude indexing of BMP pictures
Line 389
Disallow *.fla Thumbs.db *.rpm *.rm *.qt *.mov *.pfb *.ttf* *.fon *.mpg *.mpeg *.ai
    Append: *.wmv   # to exclude indexing of Windows Video files
Line 390
Disallow *.t3x *.t3d *.dll *.dat *.cab *.inf
    Append: *.98.html

List of pages in this Category

-- RudolfReuter 2010-03-18 19:19:23

Go Back to CategoryTypo3 or FrontPage ; KontaktEmail (ContactEmail)