http://creativecommons.org/licenses/by-sa/2.5/ http://swik.net/Xapian Xapian <p>Xapian is a full text search engine. It is written in <a class="wikilink" href="http://swik.net/C%2B%2B">C++ </a>, but has all the bindings you would probably want:</p> <ul> <li><span class="caps">SWIG</span> bindings. Tested for: <a class="wikilink" href="http://swik.net/Python">Python</a>, <a class="wikilink" href="http://swik.net/PHP"><span class="caps">PHP</span></a>, <a class="wikilink" href="http://swik.net/TCL"><span class="caps">TCL</span></a>, <a class="wikilink" href="http://swik.net/C%23">C#</a>, and <a class="wikilink" href="http://swik.net/Ruby">Ruby</a></li> <li>Java <a class="wikilink" href="http://swik.net/JNI"><span class="caps">JNI</span></a> bindings are included in the xapian-bindings module</li> <li><a class="wikilink" href="http://swik.net/Perl">Perl</a> XS bindings are available from <a class="wikilink" href="http://swik.net/CPAN"><span class="caps">CPAN</span></a> in module Search::Xapian</li> </ul> <h2>Bindings</h2> <ul> <li><a class="wikilink" href="http://swik.net/Xapwrap">Xapwrap</a> &#8211; a Python binding for Xapian</li> </ul> <h2>See Also</h2> <ul> <li><a class="wikilink" href="http://swik.net/Lucene">Lucene</a> &#8211; a Java Apache search engine</li> </ul> Xapian is a full text search engine. It is written in C++ , but has all the bindings you would probably want: SWIG bindings. Tested for: Python, PHP, TCL, C#, and Ruby Java JNI bindings are included in the xapian-bindings module Perl XS bindings are available from CPAN in module Search::Xapian Bindings Xapwrap &#8211; a Python binding for Xapian See Also Lucene &#8211; a Java Apache search engine http://swik.net/Xapian xapian search Indexing/Search search-engine License:GPL Wed, 06 Jul 2005 08:34:40 -0700 Sat, 13 Jan 2007 04:59:02 -0800 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Search+delay+with+new+version+of+Xapian+1.0.8/cheku We're going to need more detail than that. A test case would be ideal, but failing that: Is the index you are searching with 1.0.8 created by 1.0.7 or 1.0.8? Are you using quartz or flint? How can you tell that the search delay is "just before the query starts"? A prompt reply would be good, as 1.0.9 is nearly ready for release... Given 1.0.8 is mostly bug-fixes and a couple of them are quite important (e.g. bug #287 can cause database corruption), I think that's somewhat dubious advice. Cheers, Olly Sat, 11 Oct 2008 17:52:01 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Search+delay+with+new+version+of+Xapian+1.0.8/chegn After installing Xapian version version 1.0.8 ... I noticed search delay, just before the query starts. The delay is very much noticeable when search starts, and could be related either connecting to index database or parsing Xapian query. I test this issue on various platform including FastCGI, C++, Perl and quest command prompt program that come with Xapian and all showing short delay before any search starts. When using Xapian 1.0.7 on the same same data indexed by Xapian 1.0.7 the search delay does not exist. PS: I do not recommend to upgrade to Xapian 1.0.8 Thanks, Kevin Duraj http://pacificair.com Sat, 11 Oct 2008 16:54:00 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Problem+with+stemming+and+prefixes./chd4q Indeed that works with add_boolean_prefix() ! The "ucfirst"-tip works too but I think it's better to use the appropriate function ! Thanks to both of you. Ivan 2008/10/10 Olly Betts survex.com> Sat, 11 Oct 2008 13:57:05 -0700 http://swik.net/opensource/del.icio.us+tag%2Fopensource/The+Xapian+Project/chaok Fri, 10 Oct 2008 09:04:46 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Stemming+prefix/cg956 Hi, Is there a change to change the default-prefix ("Z") for stemmed terms? Thanks a lot! Markus Fri, 10 Oct 2008 06:08:47 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Making+SORTAFTER+useful+in+omega%3F/cg9bk We do know that indeed. It depends on the product's category. Perhaps you could allow to somehow script/configure extra behaviour, a bit like all those java-solutions which can load extra modules/classes/etc based on some (xml)configuration file. Then you could configure that you'd want some age based posting source to be active as soon as a flag F is supplied which is set to give items from last week extra weight W1, items from at most half a year ago weight W2, etc. Another useful posting source would be one that can decide to use some value or based on the existence of some term/boolean to give extra weight. Best regards, Arjen Fri, 10 Oct 2008 01:05:16 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Making+SORTAFTER+useful+in+omega%3F/cg8x4 If you have a way of deciding which results want the boost, then yes. You'll need to be able to decide that for any solution to work though. It's not much harder to use than the Sorter. But both suffer from requiring source code changes for use in Omega, which is a problem for all user subclasses. I'm not sure what the best answer is - at one point it was suggested to rewrite Omega in a scripting language (e.g. Python) but that's quite a job. You could also turn Omega into an API which could be wrapped with SWIG, but that's also a fair amount of work. Both of these approaches also mean that you incur the overhead of a trip from C++ to Python and back for each operation, which can add up when you do it for a million documents. Allowing plugins to Omega would mean you'd not need to recompile the whole of Omega, just the plugin, but I'm not sure that really makes it significantly less hassle. Cheers, Olly Thu, 09 Oct 2008 22:00:50 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Problem+with+stemming+and+prefixes./cg8jj You want add_boolean_prefix() here. add_prefix() is for "free text" like fields. Cheers, Olly Thu, 09 Oct 2008 19:01:41 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+xapian-compact+and+%22Bad+line+1+in+stubdatabase+file%22/cg71g Are you perhaps trying to open "index.compact/postlist.baseA" as a database (rather than "index.compact")? That's OK - after compaction, the "A" revision is for an empty table, the "B" revision is for the table with all the data added. If it's not the database path, please try 1.0.8. Cheers, Olly Thu, 09 Oct 2008 18:05:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+xapian-compact+and+%22Bad+line+1+in+stubdatabase+file%22/cg71f I'm pretty sure I am using the directory. [...] Okay, I will do that and report what happens. Thanks! Best regards, Adam Thu, 09 Oct 2008 18:05:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Problem+with+stemming+and+prefixes./cg71e Hi there, I am encountering problems with stemming and prefixes. I want to filter my queries with prefixes using add_term and add_prefix (as I've seen it's the best solution). So, here is the situation : my database contains movie titles, movie categories and production years. I do not filter the titles, but I filter the categories and the years, using : $doc->add_term("XYEAR".mysql_result(/* request to the year's field */)); $doc->add_term("XCAT".strtolower(mysql_result(/* request to the category's field */))); Note that the strtolower() is not the source of the problem and it is mandatory. Then, I set the add_term (I stop with the "year" case because it works) : $queryparser->add_prefix("cat", "XCAT"); Then I call my request through Firefox : http://coolurl/search.php?myssearch=cat:Thriller That works. It returns all the thriller movies. But this : http://coolurl/search.php?myssearch=cat:thriller Do not work. Indeed, a "print $query->get_description();" tells me that the queries are respectively : X Thu, 09 Oct 2008 18:05:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Problem+with+stemming+and+prefixes./cg71d As you know that it is a cat search, from your prefix "cat", the simplest solution is to upper case the first letter using the function ucfirst. Thu, 09 Oct 2008 18:05:14 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/xapian-compact+and+%22Bad+line+1+in+stub+databasefile%22/cg6hf I have a xapian index on a test-machine that has been used quite a bit. The size of the index on disk is 2.4GB, so I thought I'd run xapian-compact on it, to make it more likely to fit in RAM. It went like this: $ time xapian-compact index index.compact postlist: Reduced by 50.0176% 387080K (773888K -> 386808K) record: Reduced by 40.8424% 5120K (12536K -> 7416K) termlist: Reduced by 42.869% 62064K (144776K -> 82712K) position: Reduced by 44.1926% 651760K (1474816K -> 823056K) value: Reduced by 47.2156% 6376K (13504K -> 7128K) spelling: Size unchanged (0K) synonym: Size unchanged (0K) real 7m18.317s user 0m46.211s sys 0m10.477s $ A nice reduction to 1.3GB. But when I try to use the new index in place of the non-compact one, it doesn't work, the application says: Exception: Bad line 1 in stub database file `index.compact/postlist.baseA' at /usr/lib/perl5/Search/Xapian/Database.pm line 48 If I look at the file-size: -rw-rw-r-- 1 adsj staff 13 2008-10-08 20:01 index.compact/ Wed, 08 Oct 2008 11:52:40 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+mistake+inhttp%3A%2F%2Fxapian.org%2Fdocs%2Fvalueranges.html/cg5vr It should indeed - now fixed in SVN. Thanks. And this explicit empty destructor isn't needed, so I've removed it. Cheers, Olly Wed, 08 Oct 2008 08:51:58 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/mistake+in+http%3A%2F%2Fxapian.org%2Fdocs%2Fvalueranges.html/cg4fn I think there is a mistake in http://xapian.org/docs/valueranges.html, SECTION "Custom subclasses", source followed: struct AuthorValueRangeProcessor : public Xapian::ValueRangeProcessor { AuthorValueRangeProcessor() {} ~AuthorValueRangeProcessor() {} Xapian::valueno operator()(std::string &begin, std::string &end) { if (begin.substr(0, 7) != "author:") return Xapian::BAD_VALUENO; begin.erase(0, 7); begin = Xapian::Unicode::tolower(term); end = Xapian::Unicode::tolower(term); return 4; } }; term is the parameter of tolower in it's definition: begin = Xapian::Unicode::tolower(term); end = Xapian::Unicode::tolower(term); shoule be : begin = Xapian::Unicode::tolower(begin); end = Xapian::Unicode::tolower(end); Wed, 08 Oct 2008 01:53:23 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cg0lx Please read the website before asking further questions. You're wasting a considerable amount of the developers time, which could be better spent improving Xapian. The answer in this case is to be found on the "bleeding edge" page, linked to from the front page of the website. Or, you could have searched for "svn" in the search box on the front page of the website, which would have returned that page at the top of the result list. Or, you could have searched on Google for "xapian subversion", which would have returned a mailing list message pointing to the web interface to svn. I won't answer any further questions which are this easy to find the answers to. Tue, 07 Oct 2008 03:52:14 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cg0fk If you just want the source for xapian, it's the first link on http://xapian.org/download.php If you're wanting the source for the Debian package, you need to use the "apt-get source" command: probably "apt-get source libxapian15" (libxapian15 is the most recent Xapian library package). This is a basic Debian usage question, rather than a Xapian question, though - this mailing list isn't an appropriate place to ask for help with using apt. Tue, 07 Oct 2008 02:50:29 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cg0fj Is there any CVS or SVN of xapian? 2008/10/7 Richard Boulton lemurconsulting.com> Tue, 07 Oct 2008 02:50:29 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cgz1o I solved it by modifing source.list, change stable to etch : deb http://www.xapian.org/debian etch main deb-src http://www.xapian.org/debian etch main 2008/10/7 Olly Betts survex.com> Tue, 07 Oct 2008 00:51:51 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cgz1n Which is source, i see none of them : dndeb:~# apt-cache search xapian libxapian13 - Search engine library strigi-daemon - fast indexing and searching tool for your personal data (daemon) libxapian-dev - Development files for Xapian search engine library libxapian-ruby1.8 - Xapian search engine interface for Ruby 1.8 libxapian15 - Search engine library php4-xapian - Xapian search engine interface for PHP4 php5-xapian - Xapian search engine interface for PHP5 python-xapian - Xapian search engine interface for Python tclxapian - Xapian search engine interface for Tcl xapian-examples - Xapian simple example programs xapian-omega - CGI search interface and indexers using Xapian xapian-tools - Basic tools for Xapian search engine library xapian-doc - Core Xapian documentation 2008/10/7 Allen gmail.com> Tue, 07 Oct 2008 00:51:51 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+trouble+on+getting+source/cgzlx No file is created locally - wget is downloading the file to stdout, which is piped into apt-key's stdin. The warning is harmless really (etch *is* the current debian stable), but you can fix it by replacing "stable" with "etch" in that line in "/etc/apt/sources.list" if you want. Um, if you're running stable, don't try to use packages from unstable - they are likely not to work properly. If you are running unstable, then sid *is* debian unstable, but you can explicitly write "sid" in "/etc/apt/sources.list" if you prefer. Cheers, Olly Mon, 06 Oct 2008 21:56:32 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/trouble+on+getting+source/cgy72 I have a problem on getting source of xapian. 1st, run 'wget -O- http://www.xapian.org/debian/archive_key.asc|apt-key add -', result shows '09:35:29 (6.42 KB/s) - `-' saved [2437/2437]' and OK, but i found nothing on my disk, 2nd, run 'apt-get update' but get a warnning: W: Conflicting distribution: http://www.xapian.org stable Release (expected stable but got etch) If it caused by the 1st step? I modify my /etc/apt/sources.list, from stable to unstable: deb http://www.xapian.org/debian unstable main deb-src http://www.xapian.org/debian unstable main run 'apt-get update' but get another warnning: W: Conflicting distribution: http://www.xapian.org unstable Release (expected unstable but got sid) Mon, 06 Oct 2008 18:54:30 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgws8 OK, I've found the cause. The document lengths mean that we get weights, but if you test on a large messy database with a lot of documents with length 1, then the top 10 all have the same high weight and you get all 100% entries in the MSet (which is why neither Richard nor I could reproduce when we tried). The attached patch fixes this for me. I'll distill your example into a testcase and commit it. Cheers, Olly _______________________________________________ Xapian-discuss mailing list Xapian-discusslists.xapian.org http://lists.xapian.org/mailman/listinfo/xapian-discuss Mon, 06 Oct 2008 05:48:06 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwjd $ ./quickstartindex proverbs/ "hello" a $ ./quickstartindex proverbs/ "world" a b $ ./quickstartsearch proverbs/ a Performing query 'Xapian::Query(a)' 2 results found #1 100% [hello] #2 84% [world] $ ./quickstartsearchall proverbs/ 2 results found #1 100% [hello] #2 84% [world] 2008/10/6 Richard Boulton lemurconsulting.com> Mon, 06 Oct 2008 04:29:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwjc And the source code to quickstartsearchall is? Cheers, Olly Mon, 06 Oct 2008 04:29:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwjb The sources come from http://www.xapian.org/docs/quickstart.html, and in quickstartsearchall, like quickstartsearch, but query replaced by Query::MatchAll 2008/10/6 Olly Betts survex.com> Mon, 06 Oct 2008 04:29:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwja int main(int argc, char *argv[]) { if (argc survex.com> Mon, 06 Oct 2008 04:29:15 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwi9 Since you actually have the source file which demonstrates this problem on your hard disk, please save me the time having to glue together the code from that document and try to recreate the change you made - just attach your source code to your reply. You know, like I asked you to do several messages ago... If I have to play "guess the example code" that wastes my time, and it also wastes yours when my attempt works for me and I need to ask you for more details (like it did in this case when I tried to recreate your problem by modifying quest.cc). Tracking down bugs can be a frustrating activity, especially when you can't reproduce an issue someone else can because they simply aren't providing enough information, even when it would actually be easier for them to do so than keep sending unhelpfully vague replies. Simon Tatham's essay on the subject is well worth a read if you don't understand my frustration: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html Cheers, Olly Mon, 06 Oct 2008 04:29:14 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwi8 O.K. here is the quickstartindex.cpp #include #include using namespace std; int main(int argc, char *argv[]) { if (argc #include using namespace std; using namespace Xapian; int main(int argc, char *argv[]) { if (argc Mon, 06 Oct 2008 04:29:14 -0700 http://swik.net/Xapian/Xapian+General+Mailing+List/Re%3A+Query%3A%3AMatchAll/cgwi7 Sigh. We're getting closer. But that's not the *complete compilable* example I'm after - it's missing all the "#include" directives, and perhaps other required stuff. What I'm after is something I can compile and then run to recreate this problem. Then I can poke it with debugging tools to work out what's going on. And if your example works for me, I can be sure that it's not because we're using different code so we can move on to looking how our environments differ. I'm *NOT* just after some code to read. It would actually be simpler for you to attach the full source file than trim it down. And it would save me the tedious job of having to recreate all the lines you carefully chopped out. Cheers, Olly Mon, 06 Oct 2008 04:29:14 -0700