diff --git a/docs/admin/system_admin/indexing.rst b/docs/admin/system_admin/indexing.rst index edd9477b..8d6fda99 100644 --- a/docs/admin/system_admin/indexing.rst +++ b/docs/admin/system_admin/indexing.rst @@ -3,7 +3,6 @@ Full-text Search ---------------- - RhodeCode provides a full text search capabilities to search inside file content, commit message, and file paths (This feature is only available in Enterprise edition). Indexing is not enabled by default and to use full text search building an index is a pre-requisite. @@ -11,310 +10,11 @@ Indexing is not enabled by default and to use full text search building an index |RCE| provides support for `ElasticSearch 8`_ as a backend to index |repos|. See :ref:`enable-elasticsearch` for details. - -Indexing -^^^^^^^^ - -To run the indexer you need to have an generated |authtoken| with admin rights to all |repos|. - -To index repositories stored in RhodeCode, you have the option to set the indexer up in a -number of ways, for example: - -* Call the indexer via a cron job. We recommend running this once at night. - In case you need everything indexed immediately it's possible to index few - times during the day. Indexer has a special locking mechanism that won't allow - two instances of indexer running at once. It's safe to run it even every 1hr. -* Set the indexer to infinitely loop and reindex as soon as it has run its previous cycle. -* Hook the indexer up with your CI server to reindex after each push. - -The indexer works by indexing new commits added since the last run, and comparing -file changes to index only new or modified files. -If you wish to build a brand new index from scratch each time, use the ``force`` -option in the configuration file, or run it with --force flag. - -.. important:: - - You need to have |RCT| installed, see :ref:`install-tools`. Since |RCE| - 3.5.0 they are installed by default and available with community/enterprise installations. - -To set up indexing, use the following steps: - -1. :ref:`config-rhoderc`, if running tools remotely. -2. :ref:`run-index` -3. :ref:`set-index` -4. :ref:`advanced-indexing` - -.. _config-rhoderc: - -Configure the ``.rhoderc`` File -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - .. note:: - Optionally it's possible to use indexer without the ``.rhoderc``. Simply instead of - executing with `--instance-name=rcstack-instance` execute providing the host and token - directly: `--api-host=http://127.0.0.1:10000 --api-key=` - - -|RCT| uses the :file:`/home/{user}/.rhoderc` file for connection details -to |RCE| instances. If this file is not automatically created, -you can configure it using the following example. You need to configure the -details for each instance you want to index. - -.. code-block:: bash - - - # Get the status of each instance you wish to use with Tools - $ ./rcstack status - - RCSTACK: v5.27.3, running hostname: http://code.rhodecode.com - - -To get your API Token, on the |RCE| interface go to -:menuselection:`username --> My Account --> Auth tokens` - - -Given we're running rcstack, lets create the configuration file using docker mounts - -.. code-block:: bash - - ./rcstack cli cmd rhodecode-tools setup-config \ - --filename=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - api_host=https://code.rhodecode.com,api_key=secret-api-key - - -Let's verify created config using list-instances command - -.. code-block:: bash - - ./rcstack cli cmd rhodecode-tools list-instances --config=/vol/confvolume/.rhoderc - *Volumes*: - attaching /mnt/mac/Users/marcin/workspace/rhodecode-enterprise-docker/config/_shared under: /vol/confvolume - - ENTRYPOINT: Running rhodecode_vcsserver with cmd 'rhodecode-tools' - checking if config files needs bootstrapping - - [instance:rcstack-instance] - Config only - API-HOST: https://code.rhodecode.com - API-KEY: secret-api-key - - -This is how the file should look like - -.. code-block:: ini - - # Configure .rhoderc with matching details - # This allows the indexer to connect to the instance - [instance:rcstack-instance] - api_host = http://code.rhodecode.com - api_key = secret-api-key - - - -.. _run-index: - -Run the Indexer -^^^^^^^^^^^^^^^ - -Run the indexer using the following command, and specify the instance you want to index against: - -.. code-block:: bash - - # Using default simples indexing of all repositories - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --engine-location=http://elasticsearch-host:9200 - - # Create a custom mapping (search_mapping.ini) rules file for detailed indexing invocation - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --create-mapping=/vol/confvolume/search_mapping.ini - - # Using a custom mapping file with indexing rules - ./rcstack cli cmd rhodecode-index \ - --config=/vol/confvolume/.rhoderc \ - --instance-name=rcstack-instance \ - --engine-location=http://elasticsearch-host:9200 \ - --mapping=/vol/confvolume/search_mapping.ini - - -.. _set-index: - -Schedule the Indexer -^^^^^^^^^^^^^^^^^^^^ - -To schedule the indexer, configure the crontab file to run the indexer inside -your |RCT| virtualenv using the following steps. - -1. Open the crontab file, using ``crontab -e``. -2. Add the indexer to the crontab, and schedule it to run as regularly as you - wish. -3. Save the file. - -.. code-block:: bash - - $ crontab -e - - # The virtualenv can be called using its full path, so for example you can - # put this example into the crontab - - # Run the indexer daily at 4am using the default mapping settings - * 4 * * * - - # Run the indexer every Sunday at 3am using default mapping - * 3 * * 0 - - # Run the indexer every 15 minutes - # using a specially configured mapping file - */15 * * * * - -.. _advanced-indexing: - -Advanced Indexing -^^^^^^^^^^^^^^^^^ - - -Force Re-Indexing single repository -+++++++++++++++++++++++++++++++++++ - -Often it's required to re-index whole repository because of some repository changes, -or to remove some indexed secrets, or files. There's a special `--repo-name=` flag -for the indexer that limits execution to a single repository. For example to force-reindex -single repository such call can be made:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --force --repo-name=rhodecode-vcsserver - - -Removing repositories from index -++++++++++++++++++++++++++++++++ - -The indexer automatically removes renamed repositories and builds index for new names. -In the same way if a listed repository in mapping.ini is not reported existing by the -server it's removed from the index. -In case that you wish to remove indexed repository manually such call would allow that:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --remove-only --repo-name=rhodecode-vcsserver - - -Using search_mapping.ini file for advanced index rules -++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -By default rhodecode-index runs for all repositories, all files with parsing limits -defined by the CLI default arguments. You can change those limits by calling with -different flags such as `--max-filesize=2048kb` or `--repo-limit=10` - -For more advanced execution logic it's possible to use a configuration file that -would define detailed rules which repositories and how should be indexed. - - -.. note:: - - If you need to create the :file:`search_mapping.ini` file manually, use the |RCT| - ``./rcstack cli cmd rhodecode-index --create-mapping=path/to/search_mapping.ini`` call. - For details, see the :ref:`tools-cli` section. - -To Run the indexer with mapping file provide it using `--mapping` flag:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --mapping=/my/path/search_mapping.ini - - -Here's a detailed example of using :file:`search_mapping.ini` file. - -.. code-block:: ini - - [__DEFAULT__] - ; Create index on commits data, and files data in this order. Available options - ; are `commits`, `files` - index_types = commits,files - - ; Commit fetch limit. In what amount of chunks commits should be fetched - ; via api and parsed. This allows server to transfer smaller chunks and be less loaded - commit_fetch_limit = 1000 - - ; Commit process limit. Limit the number of commits indexer should fetch, and - ; store inside the full text search index. eg. if repo has 2000 commits, and - ; limit is 1000, on the first run it will process commits 0-1000 and on the - ; second 1000-2000 commits. Help reduce memory usage, default is 50000 - ; (set -1 for unlimited) - commit_process_limit = 20000 - - ; Limit of how many repositories each run can process, default is -1 (unlimited) - ; in case of 1000s of repositories it's better to execute in chunks to not overload - ; the server. - repo_limit = -1 - - ; Default patterns for indexing files and content of files. Binary files - ; are skipped by default. - - ; Add to index those comma separated files; globs syntax - ; e.g index_files = *.py, *.c, *.h, *.js - index_files = *, - - ; Do not add to index those comma separated files, this excludes - ; both search by name and content; globs syntax - ; e.g index_files = *.key, *.sql, *.xml, *.pem, *.crt - skip_files = , - - ; Add to index content of those comma separated files; globs syntax - ; e.g index_files = *.h, *.obj - index_files_content = *, - - ; Do not add to index content of those comma separated files; globs syntax - ; Binary files are not indexed by default. - ; e.g index_files = *.min.js, *.xml, *.dump, *.log, *.dump - skip_files_content = , - - ; Force rebuilding an index from scratch. Each repository will be rebuild from - ; scratch with a global flag. Use --repo-name=NAME --force to rebuild single repo - force = false - - ; maximum file size that indexer will use, files above that limit are not going - ; to have they content indexed. - ; Possible options are KB (kilobytes), MB (megabytes), eg 1MB or 1024KB - max_filesize = 10MB - - - [__INDEX_RULES__] - ; Ordered match rules for repositories. A list of all repositories will be fetched - ; using API and this list will be filtered using those rules. - ; Syntax for entry: `glob_pattern_OR_full_repo_name = 0 OR 1` where 0=exclude, 1=include - ; When this ordered list is traversed first match will return the include/exclude marker - ; For example: - ; upstream/binary_repo = 0 - ; upstream/subrepo/xml_files = 0 - ; upstream/* = 1 - ; special-repo = 1 - ; * = 0 - ; This will index all repositories under upstream/*, but skip upstream/binary_repo - ; and upstream/sub_repo/xml_files, last * = 0 means skip all other matches - - - ; == EXPLICIT REPOSITORY INDEXING == - ; If defined this will skip using __INDEX_RULES__, and will not use API to fetch - ; list of repositories, it will explicitly take names defined with [NAME] format and - ; try to build the index, to build index just for repo_name_1 and special-repo use: - ; [repo_name_1] - ; [special-repo] - - ; == PER REPOSITORY CONFIGURATION == - ; This allows overriding the global configuration per repository. - ; example to set specific file limit, and skip certain files for repository special-repo - ; the CLI flags doesn't override the conf settings. - ; [conf:special-repo] - ; max_filesize = 5mb - ; skip_files = *.xml, *.sql - - - -In case of 1000s of repositories it can be tricky to write the include/exclude rules at first. -There's a special flag to test the mapping file rules and list repositories that would -be indexed. Run the indexer with `--show-matched-repos` to list only the -match repositories defined in .ini file rules:: - - ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --show-matched-repos --mapping=/my/path/search_mapping.ini + For comprehensive full-text search documentation including ElasticSearch setup, + auth token generation, advanced mapping configuration, and deployment-specific instructions, + see the :ref:`Full-text Search Setup Guide `. diff --git a/docs/tools/tools-cli.rst b/docs/tools/tools-cli.rst index 3d5eac1b..4280671c 100644 --- a/docs/tools/tools-cli.rst +++ b/docs/tools/tools-cli.rst @@ -18,11 +18,11 @@ rhodecode-list-instances ------------------------ Use this command to list the instance details configured in the -:file:`/etc/rhodecode/conf/.rhoderc` file. +:file:`/vol/confvolume/.rhoderc` file when using rcstack CLI commands. .. code-block:: bash - $ ./rcstack cli cmd rhodecode-list-instances --config=/etc/rhodecode/conf/.rhoderc + $ ./rcstack cli cmd rhodecode-list-instances --config=/vol/confvolume/.rhoderc [instance:production] - Config only API-HOST: https://some.url.com API-KEY: some.auth.token @@ -231,7 +231,7 @@ Example usage: repo_dir = /var/opt/rhodecode_repo_store # Run rcstack cli - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... @@ -242,14 +242,14 @@ Example usage: are you sure you want to remove them? [y/N]: # Clean up repos older than 1 year - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --older-than=365d + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --older-than=365d checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... preparing to remove [343] found repositories older than 365 days # clean up repos older than 3 days - ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --older-than=3d + ./rcstack cli cmd rhodecode-cleanup-repos --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --older-than=3d checking if config files needs bootstrapping Scanning for repositories in /var/opt/rhodecode_repo_store... @@ -449,7 +449,7 @@ Example usage: # Cat a file and pipe to gist - $ cat ~/.rhoderc | ./rcstack cli cmd rhodecode-gist --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc -d '.rhoderc copy' create + $ cat ~/.rhoderc | ./rcstack cli cmd rhodecode-gist --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc -d '.rhoderc copy' create { "error": null, "id": 9253, @@ -540,10 +540,10 @@ Example usage: .. code-block:: bash # Create the indexing mapping file - $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc --create-mapping search_mapping.ini + $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc --create-mapping search_mapping.ini # Run the indexer - $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/etc/rhodecode/conf/.rhoderc + $ ./rcstack cli cmd rhodecode-index --instance-name=rcstack-instance --config=/vol/confvolume/.rhoderc # Run indexer based on search_mapping.ini file using rhodecode-tools virtualenv (venv)$ rhodecode-index --instance-name=rcstack-instance @@ -552,6 +552,11 @@ Example usage: # the .rhoderc file $ rhodecode-index --apikey=key --apihost=http://rhodecode.server --instance-name=rcstack-instance --save-config +.. note:: + + For comprehensive indexing documentation including ElasticSearch setup, advanced mapping configuration, + and troubleshooting, see the :ref:`Full-text Search Setup Guide `. + .. _tools-setup-config: