docs: updated indexing section
This commit is contained in:
RhodeCode Admin
parent
ef74fe98e9
commit
87e343ab83
2 changed files with 67 additions and 95 deletions
|
|
@ -3,14 +3,9 @@
|
|||
Full-text Search
|
||||
----------------
|
||||
|
||||
.. important::
|
||||
|
||||
Some parts of this page refers to legacy components like rccontrol
|
||||
We are in process of updating documentantion with up-to-date information
|
||||
|
||||
|
||||
RhodeCode provides a full text search capabilities to search inside file content,
|
||||
commit message, and file paths (This featureis only available in Enterprise edition).
|
||||
commit message, and file paths (This feature is only available in Enterprise edition).
|
||||
Indexing is not enabled by default and to use full text search building an index is a pre-requisite.
|
||||
|
||||
|RCE| provides support for `ElasticSearch 8`_ as a backend to index |repos|.
|
||||
|
|
@ -20,7 +15,7 @@ See :ref:`enable-elasticsearch` for details.
|
|||
Indexing
|
||||
^^^^^^^^
|
||||
|
||||
To run the indexer you need to have an |authtoken| with admin rights to all |repos|.
|
||||
To run the indexer you need to have an generated |authtoken| with admin rights to all |repos|.
|
||||
|
||||
To index repositories stored in RhodeCode, you have the option to set the indexer up in a
|
||||
number of ways, for example:
|
||||
|
|
@ -57,7 +52,7 @@ Configure the ``.rhoderc`` File
|
|||
.. note::
|
||||
|
||||
Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of
|
||||
executing with `--instance-name=enterprise-1` execute providing the host and token
|
||||
executing with `--instance-name=rcstack-instance` execute providing the host and token
|
||||
directly: `--api-host=http://127.0.0.1:10000 --api-key=<auth-token-goes-here>`
|
||||
|
||||
|
||||
|
|
@ -68,26 +63,53 @@ details for each instance you want to index.
|
|||
|
||||
.. code-block:: bash
|
||||
|
||||
# Check the instance details
|
||||
# of the instance you want to index
|
||||
$ rccontrol status
|
||||
|
||||
- NAME: enterprise-1
|
||||
- STATUS: RUNNING
|
||||
- TYPE: Enterprise
|
||||
- VERSION: 4.1.0
|
||||
- URL: http://127.0.0.1:10003
|
||||
# Get the status of each instance you wish to use with Tools
|
||||
$ ./rcstack status
|
||||
|
||||
RCSTACK: v5.27.3, running hostname: http://code.rhodecode.com
|
||||
|
||||
|
||||
To get your API Token, on the |RCE| interface go to
|
||||
:menuselection:`username --> My Account --> Auth tokens`
|
||||
|
||||
|
||||
Given we're running rcstack, lets create the configuration file using docker mounts
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
./rcstack cli cmd rhodecode-tools setup-config \
|
||||
--filename=/vol/confvolume/.rhoderc \
|
||||
--instance-name=rcstack-instance \
|
||||
api_host=https://code.rhodecode.com,api_key=secret-api-key
|
||||
|
||||
|
||||
Let's verify created config using list-instances command
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
./rcstack cli cmd rhodecode-tools list-instances --config=/vol/confvolume/.rhoderc
|
||||
*Volumes*:
|
||||
attaching /mnt/mac/Users/marcin/workspace/rhodecode-enterprise-docker/config/_shared under: /vol/confvolume
|
||||
|
||||
ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-tools'
|
||||
checking if config files needs bootstrapping
|
||||
|
||||
[instance:rcstack-instance] - Config only
|
||||
API-HOST: https://code.rhodecode.com
|
||||
API-KEY: secret-api-key
|
||||
|
||||
|
||||
This is how the file should look like
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
# Configure .rhoderc with matching details
|
||||
# This allows the indexer to connect to the instance
|
||||
[instance:enterprise-1]
|
||||
api_host = http://127.0.0.1:10000
|
||||
api_key = <auth token goes here>
|
||||
[instance:rcstack-instance]
|
||||
api_host = http://code.rhodecode.com
|
||||
api_key = secret-api-key
|
||||
|
||||
|
||||
|
||||
.. _run-index:
|
||||
|
|
@ -95,35 +117,28 @@ To get your API Token, on the |RCE| interface go to
|
|||
Run the Indexer
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
Run the indexer using the following command, and specify the instance you want to index:
|
||||
Run the indexer using the following command, and specify the instance you want to index against:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Using default simples indexing of all repositories
|
||||
$ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
|
||||
--instance-name=enterprise-1
|
||||
# Using default simples indexing of all repositories
|
||||
./rcstack cli cmd rhodecode-index \
|
||||
--config=/vol/confvolume/.rhoderc \
|
||||
--instance-name=rcstack-instance \
|
||||
--engine-location=http://elasticsearch-host:9200
|
||||
|
||||
# Using a custom mapping file with indexing rules, and using elasticsearch 6 backend
|
||||
$ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
|
||||
--instance-name=enterprise-1 \
|
||||
--mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini \
|
||||
--es-version=6 --engine-location=http://elasticsearch-host:9200
|
||||
# Create a custom mapping (search_mapping.ini) rules file for detailed indexing invocation
|
||||
./rcstack cli cmd rhodecode-index \
|
||||
--config=/vol/confvolume/.rhoderc \
|
||||
--instance-name=rcstack-instance \
|
||||
--create-mapping=/vol/confvolume/search_mapping.ini
|
||||
|
||||
# Using a custom mapping file and invocation without ``.rhoderc``
|
||||
$ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \
|
||||
--api-host=http://rhodecodecode.myserver.com --api-key=xxxxx \
|
||||
--mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini
|
||||
|
||||
# From inside a virtualev on your local machine or CI server.
|
||||
(venv)$ rhodecode-index --instance-name=enterprise-1
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
In case of often indexing the index may become fragmented. Most often a result of that
|
||||
is error about `too many open files`. To fix this indexer needs to be executed with
|
||||
--optimize flag. E.g `rhodecode-index --instance-name=enterprise-1 --optimize`
|
||||
This should be executed regularly, once a week is recommended.
|
||||
# Using a custom mapping file with indexing rules
|
||||
./rcstack cli cmd rhodecode-index \
|
||||
--config=/vol/confvolume/.rhoderc \
|
||||
--instance-name=rcstack-instance \
|
||||
--engine-location=http://elasticsearch-host:9200 \
|
||||
--mapping=/vol/confvolume/search_mapping.ini
|
||||
|
||||
|
||||
.. _set-index:
|
||||
|
|
@ -147,18 +162,14 @@ your |RCT| virtualenv using the following steps.
|
|||
# put this example into the crontab
|
||||
|
||||
# Run the indexer daily at 4am using the default mapping settings
|
||||
* 4 * * * /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
|
||||
--instance-name=enterprise-1
|
||||
* 4 * * * <CMD FOR INDEXING GOES HERE>
|
||||
|
||||
# Run the indexer every Sunday at 3am using default mapping
|
||||
* 3 * * 0 /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \
|
||||
--instance-name=enterprise-1
|
||||
* 3 * * 0 <CMD FOR INDEXING GOES HERE>
|
||||
|
||||
# Run the indexer every 15 minutes
|
||||
# using a specially configured mapping file
|
||||
*/15 * * * * ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \
|
||||
--instance-name=enterprise-4 \
|
||||
--mapping=/home/user/.rccontrol/enterprise-4/search_mapping.ini
|
||||
*/15 * * * * <CMD FOR INDEXING GOES HERE>
|
||||
|
||||
.. _advanced-indexing:
|
||||
|
||||
|
|
@ -174,7 +185,7 @@ or to remove some indexed secrets, or files. There's a special `--repo-name=` fl
|
|||
for the indexer that limits execution to a single repository. For example to force-reindex
|
||||
single repository such call can be made::
|
||||
|
||||
rhodecode-index --instance-name=enterprise-1 --force --repo-name=rhodecode-vcsserver
|
||||
./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --force --repo-name=rhodecode-vcsserver
|
||||
|
||||
|
||||
Removing repositories from index
|
||||
|
|
@ -185,7 +196,7 @@ In the same way if a listed repository in mapping.ini is not reported existing b
|
|||
server it's removed from the index.
|
||||
In case that you wish to remove indexed repository manually such call would allow that::
|
||||
|
||||
rhodecode-index --instance-name=enterprise-1 --remove-only --repo-name=rhodecode-vcsserver
|
||||
./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --remove-only --repo-name=rhodecode-vcsserver
|
||||
|
||||
|
||||
Using search_mapping.ini file for advanced index rules
|
||||
|
|
@ -198,22 +209,16 @@ different flags such as `--max-filesize=2048kb` or `--repo-limit=10`
|
|||
For more advanced execution logic it's possible to use a configuration file that
|
||||
would define detailed rules which repositories and how should be indexed.
|
||||
|
||||
|RCT| provides an example index configuration file called :file:`search_mapping.ini`.
|
||||
This file is created by default during installation and is located at:
|
||||
|
||||
* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`, using default |RCT|.
|
||||
* :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`,
|
||||
when using ``virtualenv``.
|
||||
|
||||
.. note::
|
||||
|
||||
If you need to create the :file:`search_mapping.ini` file manually, use the |RCT|
|
||||
``rhodecode-index --create-mapping path/to/search_mapping.ini`` API call.
|
||||
``./rcstack cli cmd rhodecode-index --create-mapping=path/to/search_mapping.ini`` call.
|
||||
For details, see the :ref:`tools-cli` section.
|
||||
|
||||
To Run the indexer with mapping file provide it using `--mapping` flag::
|
||||
|
||||
rhodecode-index --instance-name=enterprise-1 --mapping=/my/path/search_mapping.ini
|
||||
./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --mapping=/my/path/search_mapping.ini
|
||||
|
||||
|
||||
Here's a detailed example of using :file:`search_mapping.ini` file.
|
||||
|
|
@ -309,41 +314,8 @@ There's a special flag to test the mapping file rules and list repositories that
|
|||
be indexed. Run the indexer with `--show-matched-repos` to list only the
|
||||
match repositories defined in .ini file rules::
|
||||
|
||||
rhodecode-index --instance-name=enterprise-1 --show-matched-repos --mapping=/my/path/search_mapping.ini
|
||||
./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --show-matched-repos --mapping=/my/path/search_mapping.ini
|
||||
|
||||
|
||||
.. _enable-elasticsearch:
|
||||
|
||||
Enabling ElasticSearch
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
ElasticSearch is available in EE edition only. It provides much scalable and more advanced
|
||||
search capabilities. New ElasticSearch also provides much more advanced query language.
|
||||
It allows advanced filtering by file paths, extensions, use OR statements, ranges etc.
|
||||
Please check query language examples in the search field for some advanced query language usage.
|
||||
|
||||
|
||||
1. Open the :file:`rhodecode.ini` file for the instance you wish to edit. The
|
||||
default location is
|
||||
:file:`config/_shared/rhodecode.ini`
|
||||
|
||||
2. Find the search configuration section and change it to:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
search.module = rc_elasticsearch
|
||||
search.location = http://elasticsearch:9200
|
||||
## specify Elastic Search version, 8 for latest
|
||||
search.es_version = 8
|
||||
|
||||
where ``search.location`` points to the ElasticSearch server
|
||||
by default running on port 9200.
|
||||
|
||||
Index invocation also needs change. Please provide --es-version= and
|
||||
--engine-location= parameters to define ElasticSearch server location and it's version.
|
||||
For example::
|
||||
|
||||
rhodecode-index --instance-name=enterprise-1 --es-version=8 --engine-location=http://elasticsearch:9200
|
||||
|
||||
|
||||
.. _ElasticSearch 8: https://www.elastic.co/
|
||||
|
|
|
|||
|
|
@ -168,7 +168,7 @@ VCS Server Configuration
|
|||
You can configure settings for multiple VCS Servers on your
|
||||
system using their individual configuration files. Use the following
|
||||
properties inside the configuration file to set up your system. The default
|
||||
location is :file:`home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`.
|
||||
location is :file:`config/_shared/vcsserver.ini`.
|
||||
For a more detailed explanation of the logger levers, see :ref:`debug-mode`.
|
||||
|
||||
.. rst-class:: dl-horizontal
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue