From 87e343ab836cb2fc512dc9d7dc76f2b64edfb9ac Mon Sep 17 00:00:00 2001 From: RhodeCode Admin Date: Sun, 13 Jul 2025 05:32:00 +0200 Subject: [PATCH] docs: updated indexing section --- docs/admin/system_admin/indexing.rst | 160 ++++++++++--------------- docs/admin/system_admin/vcs-server.rst | 2 +- 2 files changed, 67 insertions(+), 95 deletions(-) diff --git a/docs/admin/system_admin/indexing.rst b/docs/admin/system_admin/indexing.rst index 9a60673b..edd9477b 100644 --- a/docs/admin/system_admin/indexing.rst +++ b/docs/admin/system_admin/indexing.rst @@ -3,14 +3,9 @@ Full-text Search ---------------- -.. important:: - - Some parts of this page refers to legacy components like rccontrol - We are in process of updating documentantion with up-to-date information - RhodeCode provides a full text search capabilities to search inside file content, -commit message, and file paths (This featureis only available in Enterprise edition). +commit message, and file paths (This feature is only available in Enterprise edition). Indexing is not enabled by default and to use full text search building an index is a pre-requisite. |RCE| provides support for `ElasticSearch 8`_ as a backend to index |repos|. @@ -20,7 +15,7 @@ See :ref:`enable-elasticsearch` for details. Indexing ^^^^^^^^ -To run the indexer you need to have an |authtoken| with admin rights to all |repos|. +To run the indexer you need to have an generated |authtoken| with admin rights to all |repos|. To index repositories stored in RhodeCode, you have the option to set the indexer up in a number of ways, for example: @@ -57,7 +52,7 @@ Configure the ``.rhoderc`` File .. note:: Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of - executing with `--instance-name=enterprise-1` execute providing the host and token + executing with `--instance-name=rcstack-instance` execute providing the host and token directly: `--api-host=http://127.0.0.1:10000 --api-key=` @@ -68,26 +63,53 @@ details for each instance you want to index. .. code-block:: bash - # Check the instance details - # of the instance you want to index - $ rccontrol status - - NAME: enterprise-1 - - STATUS: RUNNING - - TYPE: Enterprise - - VERSION: 4.1.0 - - URL: http://127.0.0.1:10003 + # Get the status of each instance you wish to use with Tools + $ ./rcstack status + + RCSTACK: v5.27.3, running hostname: http://code.rhodecode.com + To get your API Token, on the |RCE| interface go to :menuselection:`username --> My Account --> Auth tokens` + +Given we're running rcstack, lets create the configuration file using docker mounts + +.. code-block:: bash + + ./rcstack cli cmd rhodecode-tools setup-config \ + --filename=/vol/confvolume/.rhoderc \ + --instance-name=rcstack-instance \ + api_host=https://code.rhodecode.com,api_key=secret-api-key + + +Let's verify created config using list-instances command + +.. code-block:: bash + + ./rcstack cli cmd rhodecode-tools list-instances --config=/vol/confvolume/.rhoderc + *Volumes*: + attaching /mnt/mac/Users/marcin/workspace/rhodecode-enterprise-docker/config/_shared under: /vol/confvolume + + ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-tools' + checking if config files needs bootstrapping + + [instance:rcstack-instance] - Config only + API-HOST: https://code.rhodecode.com + API-KEY: secret-api-key + + +This is how the file should look like + .. code-block:: ini # Configure .rhoderc with matching details # This allows the indexer to connect to the instance - [instance:enterprise-1] - api_host = http://127.0.0.1:10000 - api_key = + [instance:rcstack-instance] + api_host = http://code.rhodecode.com + api_key = secret-api-key + .. _run-index: @@ -95,35 +117,28 @@ To get your API Token, on the |RCE| interface go to Run the Indexer ^^^^^^^^^^^^^^^ -Run the indexer using the following command, and specify the instance you want to index: +Run the indexer using the following command, and specify the instance you want to index against: .. code-block:: bash - # Using default simples indexing of all repositories - $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \ - --instance-name=enterprise-1 + # Using default simples indexing of all repositories + ./rcstack cli cmd rhodecode-index \ + --config=/vol/confvolume/.rhoderc \ + --instance-name=rcstack-instance \ + --engine-location=http://elasticsearch-host:9200 - # Using a custom mapping file with indexing rules, and using elasticsearch 6 backend - $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \ - --instance-name=enterprise-1 \ - --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini \ - --es-version=6 --engine-location=http://elasticsearch-host:9200 + # Create a custom mapping (search_mapping.ini) rules file for detailed indexing invocation + ./rcstack cli cmd rhodecode-index \ + --config=/vol/confvolume/.rhoderc \ + --instance-name=rcstack-instance \ + --create-mapping=/vol/confvolume/search_mapping.ini - # Using a custom mapping file and invocation without ``.rhoderc`` - $ /home/user/.rccontrol/enterprise-1/profile/bin/rhodecode-index \ - --api-host=http://rhodecodecode.myserver.com --api-key=xxxxx \ - --mapping=/home/user/.rccontrol/enterprise-1/search_mapping.ini - - # From inside a virtualev on your local machine or CI server. - (venv)$ rhodecode-index --instance-name=enterprise-1 - - -.. note:: - - In case of often indexing the index may become fragmented. Most often a result of that - is error about `too many open files`. To fix this indexer needs to be executed with - --optimize flag. E.g `rhodecode-index --instance-name=enterprise-1 --optimize` - This should be executed regularly, once a week is recommended. + # Using a custom mapping file with indexing rules + ./rcstack cli cmd rhodecode-index \ + --config=/vol/confvolume/.rhoderc \ + --instance-name=rcstack-instance \ + --engine-location=http://elasticsearch-host:9200 \ + --mapping=/vol/confvolume/search_mapping.ini .. _set-index: @@ -147,18 +162,14 @@ your |RCT| virtualenv using the following steps. # put this example into the crontab # Run the indexer daily at 4am using the default mapping settings - * 4 * * * /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \ - --instance-name=enterprise-1 + * 4 * * * # Run the indexer every Sunday at 3am using default mapping - * 3 * * 0 /home/ubuntu/.virtualenv/rhodecode-venv/bin/rhodecode-index \ - --instance-name=enterprise-1 + * 3 * * 0 # Run the indexer every 15 minutes # using a specially configured mapping file - */15 * * * * ~/.rccontrol/enterprise-4/profile/bin/rhodecode-index \ - --instance-name=enterprise-4 \ - --mapping=/home/user/.rccontrol/enterprise-4/search_mapping.ini + */15 * * * * .. _advanced-indexing: @@ -174,7 +185,7 @@ or to remove some indexed secrets, or files. There's a special `--repo-name=` fl for the indexer that limits execution to a single repository. For example to force-reindex single repository such call can be made:: - rhodecode-index --instance-name=enterprise-1 --force --repo-name=rhodecode-vcsserver + ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --force --repo-name=rhodecode-vcsserver Removing repositories from index @@ -185,7 +196,7 @@ In the same way if a listed repository in mapping.ini is not reported existing b server it's removed from the index. In case that you wish to remove indexed repository manually such call would allow that:: - rhodecode-index --instance-name=enterprise-1 --remove-only --repo-name=rhodecode-vcsserver + ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --remove-only --repo-name=rhodecode-vcsserver Using search_mapping.ini file for advanced index rules @@ -198,22 +209,16 @@ different flags such as `--max-filesize=2048kb` or `--repo-limit=10` For more advanced execution logic it's possible to use a configuration file that would define detailed rules which repositories and how should be indexed. -|RCT| provides an example index configuration file called :file:`search_mapping.ini`. -This file is created by default during installation and is located at: - -* :file:`/home/{user}/.rccontrol/{instance-id}/search_mapping.ini`, using default |RCT|. -* :file:`~/venv/lib/python2.7/site-packages/rhodecode_tools/templates/mapping.ini`, - when using ``virtualenv``. .. note:: If you need to create the :file:`search_mapping.ini` file manually, use the |RCT| - ``rhodecode-index --create-mapping path/to/search_mapping.ini`` API call. + ``./rcstack cli cmd rhodecode-index --create-mapping=path/to/search_mapping.ini`` call. For details, see the :ref:`tools-cli` section. To Run the indexer with mapping file provide it using `--mapping` flag:: - rhodecode-index --instance-name=enterprise-1 --mapping=/my/path/search_mapping.ini + ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --mapping=/my/path/search_mapping.ini Here's a detailed example of using :file:`search_mapping.ini` file. @@ -309,41 +314,8 @@ There's a special flag to test the mapping file rules and list repositories that be indexed. Run the indexer with `--show-matched-repos` to list only the match repositories defined in .ini file rules:: - rhodecode-index --instance-name=enterprise-1 --show-matched-repos --mapping=/my/path/search_mapping.ini + ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --show-matched-repos --mapping=/my/path/search_mapping.ini -.. _enable-elasticsearch: - -Enabling ElasticSearch -^^^^^^^^^^^^^^^^^^^^^^ - -ElasticSearch is available in EE edition only. It provides much scalable and more advanced -search capabilities. New ElasticSearch also provides much more advanced query language. -It allows advanced filtering by file paths, extensions, use OR statements, ranges etc. -Please check query language examples in the search field for some advanced query language usage. - - -1. Open the :file:`rhodecode.ini` file for the instance you wish to edit. The - default location is - :file:`config/_shared/rhodecode.ini` - -2. Find the search configuration section and change it to: - -.. code-block:: ini - - search.module = rc_elasticsearch - search.location = http://elasticsearch:9200 - ## specify Elastic Search version, 8 for latest - search.es_version = 8 - -where ``search.location`` points to the ElasticSearch server -by default running on port 9200. - -Index invocation also needs change. Please provide --es-version= and ---engine-location= parameters to define ElasticSearch server location and it's version. -For example:: - - rhodecode-index --instance-name=enterprise-1 --es-version=8 --engine-location=http://elasticsearch:9200 - .. _ElasticSearch 8: https://www.elastic.co/ diff --git a/docs/admin/system_admin/vcs-server.rst b/docs/admin/system_admin/vcs-server.rst index 0a1a2728..54fc06bc 100644 --- a/docs/admin/system_admin/vcs-server.rst +++ b/docs/admin/system_admin/vcs-server.rst @@ -168,7 +168,7 @@ VCS Server Configuration You can configure settings for multiple VCS Servers on your system using their individual configuration files. Use the following properties inside the configuration file to set up your system. The default -location is :file:`home/{user}/.rccontrol/{vcsserver-id}/vcsserver.ini`. +location is :file:`config/_shared/vcsserver.ini`. For a more detailed explanation of the logger levers, see :ref:`debug-mode`. .. rst-class:: dl-horizontal