276 lines
No EOL
8.5 KiB
ReStructuredText
276 lines
No EOL
8.5 KiB
ReStructuredText
.. _indexing-ref:
|
|
|
|
Full-text Search
|
|
----------------
|
|
|
|
By default RhodeCode is configured to use `Whoosh`_ to index |repos| and
|
|
provide full-text search.
|
|
|
|
|RCE| also provides support for `Elasticsearch`_ as a backend for scalable
|
|
search. See :ref:`enable-elasticsearch` for details.
|
|
|
|
Indexing
|
|
^^^^^^^^
|
|
|
|
To run the indexer you need to use an |authtoken| with admin rights to all
|
|
|repos|.
|
|
|
|
To index new content added, you have the option to set the indexer up in a
|
|
number of ways, for example:
|
|
|
|
* Call the indexer via a cron job. We recommend running this nightly,
|
|
unless you need everything indexed immediately.
|
|
* Set the indexer to infinitely loop and reindex as soon as it has run its
|
|
cycle.
|
|
* Hook the indexer up with your CI server to reindex after each push.
|
|
|
|
The indexer works by indexing new commits added since the last run. If you
|
|
wish to build a brand new index from scratch each time,
|
|
use the ``force`` option in the configuration file.
|
|
|
|
.. important::
|
|
|
|
You need to have |RCT| installed, see :ref:`install-tools`. Since |RCE|
|
|
3.5.0 they are installed by default.
|
|
|
|
To set up indexing, use the following steps:
|
|
|
|
1. :ref:`config-rhoderc`, if running tools remotely.
|
|
2. :ref:`run-index`
|
|
3. :ref:`set-index`
|
|
4. :ref:`advanced-indexing`
|
|
|
|
.. _config-rhoderc:
|
|
|
|
Configure the ``.rhoderc`` File
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|RCT| uses the :file:`/home/{user}/.rhoderc` file for connection details
|
|
to |RCE| instances. If this file is not automatically created,
|
|
you can configure it using the following example. You need to configure the
|
|
details for each instance you want to index.
|
|
|
|
.. code-block:: bash
|
|
|
|
# Check the instance details
|
|
# of the instance you want to index
|
|
$ rccontrol status
|
|
|
|
- NAME: enterprise-1
|
|
- STATUS: RUNNING
|
|
- TYPE: Momentum
|
|
- VERSION: 1.5.0
|
|
- URL: http://127.0.0.1:10000
|
|
|
|
To get your API Token, on the |RCE| interface go to
|
|
:menuselection:`username --> My Account --> Auth tokens`
|
|
|
|
.. code-block:: ini
|
|
|
|
# Configure .rhoderc with matching details
|
|
# This allows the indexer to connect to the instance
|
|
[instance:enterprise-1]
|
|
api_host = http://127.0.0.1:10000
|
|
api_key = <auth token goes here>
|
|
repo_dir = /home/<username>/repos
|
|
|
|
.. _run-index:
|
|
|
|
Run the Indexer
|
|
^^^^^^^^^^^^^^^
|
|
|
|
Run the indexer using the following command, and specify the instance you
|
|
want to index:
|
|
|
|
.. code-block:: bash
|
|
|
|
# From inside a virtualevv
|
|
(venv)$ rhodecode-index --instance-name=enterprise-1
|
|
|
|
# Using default installation
|
|
$ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
|
|
--instance-name=enterprise-1
|
|
|
|
# Using a custom mapping file
|
|
$ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
|
|
--instance-name=enterprise-1 \
|
|
--mapping=/home/user/.rccontrol/enterprise-1/mapping.ini
|
|
|
|
.. note::
|
|
|
|
In case of often indexing the index may become fragmented. Most often a result of that
|
|
is error about `too many open files`. To fix this indexer needs to be executed with
|
|
--optimize flag. E.g `rhodecode-index --instance-name=enterprise-1 --optimize`
|
|
This should be executed regularly, once a week is recommended.
|
|
|
|
|
|
.. _set-index:
|
|
|
|
Schedule the Indexer
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
|
|
To schedule the indexer, configure the crontab file to run the indexer inside
|
|
your |RCT| virtualenv using the following steps.
|
|
|
|
1. Open the crontab file, using ``crontab -e``.
|
|
2. Add the indexer to the crontab, and schedule it to run as regularly as you
|
|
wish.
|
|
3. Save the file.
|
|
|
|
.. code-block:: bash
|
|
|
|
$ crontab -e
|
|
|
|
# The virtualenv can be called using its full path, so for example you can
|
|
# put this example into the crontab
|
|
|
|
# Run the indexer daily at 4am using the default mapping settings
|
|
* 4 * * * /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
|
|
--instance-name=enterprise-1
|
|
|
|
# Run the indexer every Sunday at 3am using default mapping
|
|
* 3 * * 0 /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
|
|
--instance-name=enterprise-1
|
|
|
|
# Run the indexer every 15 minutes
|
|
# using a specially configured mapping file
|
|
*/15 * * * * ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
|
|
--instance-name=enterprise-4 \
|
|
--mapping=/home/user/.rccontrol/enterprise-4/mapping.ini
|
|
|
|
.. _advanced-indexing:
|
|
|
|
Advanced Indexing
|
|
^^^^^^^^^^^^^^^^^
|
|
|
|
|RCT| indexes based on the :file:`mapping.ini` file. To configure your index,
|
|
you can specify different options in this file. The default location is:
|
|
|
|
* :file:`/home/{user}/.rccontrol/{instance-id}/mapping.ini`, using default
|
|
|RCT|.
|
|
* :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`,
|
|
when using ``virtualenv``.
|
|
|
|
.. note::
|
|
|
|
If you need to create the :file:`mapping.ini` file, use the |RCT|
|
|
``rhodecode-index --create-mapping path/to/file`` API call. For details,
|
|
see the :ref:`tools-cli` section.
|
|
|
|
The indexer runs in a random order to prevent a failing |repo| from stopping
|
|
a build. To configure different indexing scenarios, set the following options
|
|
inside the :file:`mapping.ini` and specify the altered file using the
|
|
``--mapping`` option.
|
|
|
|
* ``index_files`` : Index the specified file types.
|
|
* ``skip_files`` : Do not index the specified file types.
|
|
* ``index_files_content`` : Index the content of the specified file types.
|
|
* ``skip_files_content`` : Do not index the content of the specified files.
|
|
* ``force`` : Create a fresh index on each run.
|
|
* ``max_filesize`` : Files larger than the set size will not be indexed.
|
|
* ``commit_parse_limit`` : Set the batch size when indexing commit messages.
|
|
Set to a lower number to lessen memory load.
|
|
* ``repo_limit`` : Set the maximum number or |repos| indexed per run.
|
|
* ``[INCLUDE]`` : Set |repos| you want indexed. This takes precedent over
|
|
``[EXCLUDE]``.
|
|
* ``[EXCLUDE]`` : Set |repos| you do not want indexed. Exclude can be used to
|
|
not index branches, forks, or log |repos|.
|
|
|
|
At the end of the file you can specify conditions for specific |repos| that
|
|
will override the default values. To configure your indexer,
|
|
use the following example :file:`mapping.ini` file.
|
|
|
|
.. code-block:: ini
|
|
|
|
[__DEFAULT__]
|
|
# default patterns for indexing files and content of files.
|
|
# Binary files are skipped by default.
|
|
|
|
# Index python and markdown files
|
|
index_files = *.py, *.md
|
|
|
|
# Do not index these file types
|
|
skip_files = *.svg, *.log, *.dump, *.txt
|
|
|
|
# Index both file types and their content
|
|
index_files_content = *.cpp, *.ini, *.py
|
|
|
|
# Index file names, but not file content
|
|
skip_files_content = *.svg,
|
|
|
|
# Force rebuilding an index from scratch. Each repository will be rebuild
|
|
# from scratch with a global flag. Use local flag to rebuild single repos
|
|
force = false
|
|
|
|
# Do not index files larger than 385KB
|
|
max_filesize = 385KB
|
|
|
|
# Limit commit indexing to 500 per batch
|
|
commit_parse_limit = 500
|
|
|
|
# Limit each index run to 25 repos
|
|
repo_limit = 25
|
|
|
|
# __INCLUDE__ is more important that __EXCLUDE__.
|
|
|
|
[__INCLUDE__]
|
|
# Include all repos with these names
|
|
|
|
docs/* = 1
|
|
lib/* = 1
|
|
|
|
[__EXCLUDE__]
|
|
# Do not include the following repo in index
|
|
|
|
dev-docs/* = 1
|
|
legacy-repos/* = 1
|
|
*-dev/* = 1
|
|
|
|
# Each repo that needs special indexing is a separate section below.
|
|
# In each section set the options to override the global configuration
|
|
# parameters above.
|
|
# If special settings are not configured, the global configuration values
|
|
# above are inherited. If no special repositories are
|
|
# defined here RhodeCode will use the API to ask for all repositories
|
|
|
|
# For this repo use different settings
|
|
[special-repo]
|
|
commit_parse_limit = 20,
|
|
skip_files = *.idea, *.xml,
|
|
|
|
# For another repo use different settings
|
|
[another-special-repo]
|
|
index_files = *,
|
|
max_filesize = 800MB
|
|
commit_parse_limit = 20000
|
|
|
|
.. _enable-elasticsearch:
|
|
|
|
Enabling Elasticsearch
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
1. Open the :file:`rhodecode.ini` file for the instance you wish to edit. The
|
|
default location is
|
|
:file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini`
|
|
2. Find the search configuration section:
|
|
|
|
.. code-block:: ini
|
|
|
|
###################################
|
|
## SEARCH INDEXING CONFIGURATION ##
|
|
###################################
|
|
|
|
search.module = rhodecode.lib.index.whoosh
|
|
search.location = %(here)s/data/index
|
|
|
|
and change it to:
|
|
|
|
.. code-block:: ini
|
|
|
|
search.module = rc_elasticsearch
|
|
search.location = http://localhost:9200/
|
|
|
|
where ``search.location`` points to the elasticsearch server.
|
|
|
|
.. _Whoosh: https://pypi.python.org/pypi/Whoosh/
|
|
.. _Elasticsearch: https://www.elastic.co/ |