From 0583413c64126547f7256f838b6497f542348373 Mon Sep 17 00:00:00 2001 From: Andrii V Date: Wed, 30 Jul 2025 18:32:32 +0200 Subject: [PATCH] docs: removed duplicated fulltext search setup content - added link to rcstack docs instead --- docs/admin/system_admin/indexing.rst | 309 +-------------------------- docs/tools/tools-cli.rst | 21 +- 2 files changed, 19 insertions(+), 311 deletions(-) diff --git a/docs/admin/system_admin/indexing.rst b/docs/admin/system_admin/indexing.rst index edd9477b..faf21ef5 100644 --- a/docs/admin/system_admin/indexing.rst +++ b/docs/admin/system_admin/indexing.rst @@ -3,6 +3,11 @@ Full-text Search ---------------- +.. note:: + + For comprehensive full-text search documentation including ElasticSearch setup, + auth token generation, advanced mapping configuration, and deployment-specific instructions, + see the `Full-text Search Setup Guide `_. RhodeCode provides a full text search capabilities to search inside file content, commit message, and file paths (This feature is only available in Enterprise edition). @@ -12,309 +17,7 @@ Indexing is not enabled by default and to use full text search building an index See :ref:`enable-elasticsearch` for details. -Indexing -^^^^^^^^ - -To run the indexer you need to have an generated |authtoken| with admin rights to all |repos|. - -To index repositories stored in RhodeCode, you have the option to set the indexer up in a -number of ways, for example: - -* Call the indexer via a cron job. We recommend running this once at night. - In case you need everything indexed immediately it's possible to index few - times during the day. Indexer has a special locking mechanism that won't allow - two instances of indexer running at once. It's safe to run it even every 1hr. -* Set the indexer to infinitely loop and reindex as soon as it has run its previous cycle. -* Hook the indexer up with your CI server to reindex after each push. - -The indexer works by indexing new commits added since the last run, and comparing -file changes to index only new or modified files. -If you wish to build a brand new index from scratch each time, use the ``force`` -option in the configuration file, or run it with --force flag. - -.. important:: - - You need to have |RCT| installed, see :ref:`install-tools`. Since |RCE| - 3.5.0 they are installed by default and available with community/enterprise installations. - -To set up indexing, use the following steps: - -1. :ref:`config-rhoderc`, if running tools remotely. -2. :ref:`run-index` -3. :ref:`set-index` -4. :ref:`advanced-indexing` - -.. _config-rhoderc: - -Configure the ``.rhoderc`` File -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. note:: - - Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of - executing with `--instance-name=rcstack-instance` execute providing the host and token - directly: `--api-host=http://127.0.0.1:10000 --api-key=` - - -|RCT| uses the :file:`/home/{user}/.rhoderc` file for connection details -to |RCE| instances. If this file is not automatically created, -you can configure it using the following example. You need to configure the -details for each instance you want to index. - -.. code-block:: bash - - - # Get the status of each instance you wish to use with Tools - $ ./rcstack status - - RCSTACK: v5.27.3, running hostname: http://code.rhodecode.com - - -To get your API Token, on the |RCE| interface go to -:menuselection:`username --> My Account --> Auth tokens` - - -Given we're running rcstack, lets create the configuration file using docker mounts - -.. code-block:: bash - - ./rcstack cli cmd rhodecode-tools setup-config \ - --filename=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - api_host=https://code.rhodecode.com,api_key=secret-api-key - - -Let's verify created config using list-instances command - -.. code-block:: bash - - ./rcstack cli cmd rhodecode-tools list-instances --config=/vol/confvolume/.rhoderc - *Volumes*: - attaching /mnt/mac/Users/marcin/workspace/rhodecode-enterprise-docker/config/_shared under: /vol/confvolume - - ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-tools' - checking if config files needs bootstrapping - - [instance:rcstack-instance] - Config only - API-HOST: https://code.rhodecode.com - API-KEY: secret-api-key - - -This is how the file should look like - -.. code-block:: ini - - # Configure .rhoderc with matching details - # This allows the indexer to connect to the instance - [instance:rcstack-instance] - api_host = http://code.rhodecode.com - api_key = secret-api-key - - - -.. _run-index: - -Run the Indexer -^^^^^^^^^^^^^^^ - -Run the indexer using the following command, and specify the instance you want to index against: - -.. code-block:: bash - - # Using default simples indexing of all repositories - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --engine-location=http://elasticsearch-host:9200 - - # Create a custom mapping (search_mapping.ini) rules file for detailed indexing invocation - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --create-mapping=/vol/confvolume/search_mapping.ini - - # Using a custom mapping file with indexing rules - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --engine-location=http://elasticsearch-host:9200 \ - --mapping=/vol/confvolume/search_mapping.ini - - -.. _set-index: - -Schedule the Indexer -^^^^^^^^^^^^^^^^^^^^ - -To schedule the indexer, configure the crontab file to run the indexer inside -your |RCT| virtualenv using the following steps. - -1. Open the crontab file, using ``crontab -e``. -2. Add the indexer to the crontab, and schedule it to run as regularly as you - wish. -3. Save the file. - -.. code-block:: bash - - $ crontab -e - - # The virtualenv can be called using its full path, so for example you can - # put this example into the crontab - - # Run the indexer daily at 4am using the default mapping settings - * 4 * * * - - # Run the indexer every Sunday at 3am using default mapping - * 3 * * 0 - - # Run the indexer every 15 minutes - # using a specially configured mapping file - */15 * * * * - -.. _advanced-indexing: - -Advanced Indexing -^^^^^^^^^^^^^^^^^ - - -Force Re-Indexing single repository -+++++++++++++++++++++++++++++++++++ - -Often it's required to re-index whole repository because of some repository changes, -or to remove some indexed secrets, or files. There's a special `--repo-name=` flag -for the indexer that limits execution to a single repository. For example to force-reindex -single repository such call can be made:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --force --repo-name=rhodecode-vcsserver - - -Removing repositories from index -++++++++++++++++++++++++++++++++ - -The indexer automatically removes renamed repositories and builds index for new names. -In the same way if a listed repository in mapping.ini is not reported existing by the -server it's removed from the index. -In case that you wish to remove indexed repository manually such call would allow that:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --remove-only --repo-name=rhodecode-vcsserver - - -Using search_mapping.ini file for advanced index rules -++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -By default rhodecode-index runs for all repositories, all files with parsing limits -defined by the CLI default arguments. You can change those limits by calling with -different flags such as `--max-filesize=2048kb` or `--repo-limit=10` - -For more advanced execution logic it's possible to use a configuration file that -would define detailed rules which repositories and how should be indexed. - - -.. note:: - - If you need to create the :file:`search_mapping.ini` file manually, use the |RCT| - ``./rcstack cli cmd rhodecode-index --create-mapping=path/to/search_mapping.ini`` call. - For details, see the :ref:`tools-cli` section. - -To Run the indexer with mapping file provide it using `--mapping` flag:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --mapping=/my/path/search_mapping.ini - - -Here's a detailed example of using :file:`search_mapping.ini` file. - -.. code-block:: ini - - [__DEFAULT__] - ; Create index on commits data, and files data in this order. Available options - ; are `commits`, `files` - index_types = commits,files - - ; Commit fetch limit. In what amount of chunks commits should be fetched - ; via api and parsed. This allows server to transfer smaller chunks and be less loaded - commit_fetch_limit = 1000 - - ; Commit process limit. Limit the number of commits indexer should fetch, and - ; store inside the full text search index. eg. if repo has 2000 commits, and - ; limit is 1000, on the first run it will process commits 0-1000 and on the - ; second 1000-2000 commits. Help reduce memory usage, default is 50000 - ; (set -1 for unlimited) - commit_process_limit = 20000 - - ; Limit of how many repositories each run can process, default is -1 (unlimited) - ; in case of 1000s of repositories it's better to execute in chunks to not overload - ; the server. - repo_limit = -1 - - ; Default patterns for indexing files and content of files. Binary files - ; are skipped by default. - - ; Add to index those comma separated files; globs syntax - ; e.g index_files = *.py, *.c, *.h, *.js - index_files = *, - - ; Do not add to index those comma separated files, this excludes - ; both search by name and content; globs syntax - ; e.g index_files = *.key, *.sql, *.xml, *.pem, *.crt - skip_files = , - - ; Add to index content of those comma separated files; globs syntax - ; e.g index_files = *.h, *.obj - index_files_content = *, - - ; Do not add to index content of those comma separated files; globs syntax - ; Binary files are not indexed by default. - ; e.g index_files = *.min.js, *.xml, *.dump, *.log, *.dump - skip_files_content = , - - ; Force rebuilding an index from scratch. Each repository will be rebuild from - ; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo - force = false - - ; maximum file size that indexer will use, files above that limit are not going - ; to have they content indexed. - ; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB - max_filesize = 10MB - - - [__INDEX_RULES__] - ; Ordered match rules for repositories. A list of all repositories will be fetched - ; using API and this list will be filtered using those rules. - ; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include - ; When this ordered list is traversed first match will return the include/exclude marker - ; For example: - ; upstream/binary_repo = 0 - ; upstream/subrepo/xml_files = 0 - ; upstream/* = 1 - ; special-repo = 1 - ; * = 0 - ; This will index all repositories under upstream/*, but skip upstream/binary_repo - ; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches - - - ; == EXPLICIT REPOSITORY INDEXING == - ; If defined this will skip using __INDEX_RULES__, and will not use API to fetch - ; list of repositories, it will explicitly take names defined with [NAME] format and - ; try to build the index, to build index just for repo_name_1 and special-repo use: - ; [repo_name_1] - ; [special-repo] - - ; == PER REPOSITORY CONFIGURATION == - ; This allows overriding the global configuration per repository. - ; example to set specific file limit, and skip certain files for repository special-repo - ; the CLI flags doesn't override the conf settings. - ; [conf:special-repo] - ; max_filesize = 5mb - ; skip_files = *.xml, *.sql - - - -In case of 1000s of repositories it can be tricky to write the include/exclude rules at first. -There's a special flag to test the mapping file rules and list repositories that would -be indexed. Run the indexer with `--show-matched-repos` to list only the -match repositories defined in .ini file rules:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --show-matched-repos --mapping=/my/path/search_mapping.ini +For detailed indexing setup instructions, configuration examples, and troubleshooting, please refer to the comprehensive `Full-text Search Setup Guide `_. diff --git a/docs/tools/tools-cli.rst b/docs/tools/tools-cli.rst index 3d5eac1b..25a9e225 100644 --- a/docs/tools/tools-cli.rst +++ b/docs/tools/tools-cli.rst @@ -18,11 +18,11 @@ rhodecode-list-instances ------------------------ Use this command to list the instance details configured in the -:file:`/etc/rhodecode/conf/.rhoderc` file. +:file:`/vol/confvolume/.rhoderc` file when using rcstack CLI commands. .. code-block:: bash - $ ./rcstack cli cmd rhodecode-list-instances --config=/etc/rhodecode/conf/.rhoderc + $ ./rcstack cli cmd rhodecode-list-instances --config=/vol/confvolume/.rhoderc [instance:production] - Config only API-HOST: https://some.url.com API-KEY: some.auth.token @@ -231,7 +231,7 @@ Example usage: repo_dir = /var/opt/rhodecode_repo_store # Run rcstack cli - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... @@ -242,14 +242,14 @@ Example usage: are you sure you want to remove them? [y/N]: # Clean up repos older than 1 year - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --older-than=365d + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --older-than=365d checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... preparing to remove [343] found repositories older than 365 days # clean up repos older than 3 days - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --older-than=3d + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --older-than=3d checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... @@ -449,7 +449,7 @@ Example usage: # Cat a file and pipe to gist - $ cat ~/.rhoderc | ./rcstack cli cmd rhodecode-gist --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc -d '.rhoderc copy' create + $ cat ~/.rhoderc | ./rcstack cli cmd rhodecode-gist --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc -d '.rhoderc copy' create { "error": null, "id": 9253, @@ -540,10 +540,10 @@ Example usage: .. code-block:: bash # Create the indexing mapping file - $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --create-mapping search_mapping.ini + $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --create-mapping search_mapping.ini # Run the indexer - $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc + $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc # Run indexer based on search_mapping.ini file using rhodecode-tools virtualenv (venv)$ rhodecode-index --instance-name=rcstack-instance @@ -552,6 +552,11 @@ Example usage: # the .rhoderc file $ rhodecode-index --apikey=key --apihost=http://rhodecode.server --instance-name=rcstack-instance --save-config +.. note:: + + For comprehensive indexing documentation including ElasticSearch setup, advanced mapping configuration, + and troubleshooting, see the `Full-text Search Setup Guide `_. + .. _tools-setup-config: