release: merge back stable branch into default

docs/install/configure-celery.rst

@@ -4,27 +4,48 @@
 Configure Celery
 ----------------
 
-To improve |RCM| performance you should configure and enabled Celery_ as it makes
-asynchronous tasks work efficiently. Most important it allows sending notification
-emails, create repository forks, and import repositories in async way.
+Celery_ is an asynchronous task queue. It's a part of RhodeCode scheduler
+functionality. Celery_ makes certain heavy tasks perform more efficiently.
+Most important it allows sending notification emails, create repository forks,
+and import repositories in async way. It is also used for bi-directional
+repository sync in scheduler.
 
 If you decide to use Celery you also need a working message queue.
-The recommended message broker is rabbitmq_.
+The recommended and fully supported message broker is rabbitmq_.
 
 
-In order to have install and configure Celery, follow these steps:
+In order to install and configure Celery, follow these steps:
 
 1. Install RabbitMQ, see the documentation on the Celery website for
-   `rabbitmq installation`_.
+   `rabbitmq installation`_, or `rabbitmq website installation`_
 
-2. Configure Celery in the
+
+1a. As en example configuration after installation, you can run::
+
+   sudo rabbitmqctl add_user rcuser secret_password
+   sudo rabbitmqctl add_vhost rhodevhost
+   sudo rabbitmqctl set_user_tags rcuser rhodecode
+   sudo rabbitmqctl set_permissions -p rhodevhost rcuser ".*" ".*" ".*"
+
+
+2. Enable celery, and install `celery worker` process script using the `enable-module`::
+
+    rccontrol enable-module celery {instance-id}
+
+.. note::
+
+   In case when using multiple instances in one or multiple servers it's highly
+   recommended that celery is running only once, for all servers connected to
+   the same database. Having multiple celery instances running without special
+   reconfiguration could cause scheduler issues.
+
+
+3. Configure Celery in the
    :file:`home/{user}/.rccontrol/{instance-id}/rhodecode.ini` file.
-   Set the following minimal settings, that are set during rabbitmq_ installation::
+   Set the broker_url as minimal settings required to enable operation.
+   If used our example data from pt 1a, here is how the broker url should look like::
 
-        broker.host =
-        broker.vhost =
-        broker.user =
-        broker.password =
+        celery.broker_url = amqp://rcuser:secret_password@localhost:5672/rhodevhost
 
    Full configuration example is below:
 
@@ -34,35 +55,15 @@ In order to have install and configure Celery, follow these steps:
         ####################################
         ###        CELERY CONFIG        ####
         ####################################
-        ## Set to true
+
         use_celery = true
-        broker.host = localhost
-        broker.vhost = rabbitmqvhost
-        broker.port = 5672
-        broker.user = rabbitmq
-        broker.password = secret
+        celery.broker_url = amqp://rcuser:secret@localhost:5672/rhodevhost
 
-        celery.imports = rhodecode.lib.celerylib.tasks
-
-        celery.result.backend = amqp
-        celery.result.dburi = amqp://
-        celery.result.serialier = json
-
-        #celery.send.task.error.emails = true
-        #celery.amqp.task.result.expires = 18000
-
-        celeryd.concurrency = 2
-        #celeryd.log.file = celeryd.log
-        celeryd.log.level = debug
-        celeryd.max.tasks.per.child = 1
+        # maximum tasks to execute before worker restart
+        celery.max_tasks_per_child = 100
 
         ## tasks will never be sent to the queue, but executed locally instead.
-        celery.always.eager = false
-
-
-3. Enable celery, and install `celeryd` process script using the `enable-module`::
-
-    rccontrol enable-module celery {instance-id}
+        celery.task_always_eager = false
 
 
 .. _python: http://www.python.org/
@@ -70,4 +71,5 @@ In order to have install and configure Celery, follow these steps:
 .. _celery: http://celeryproject.org/
 .. _rabbitmq: http://www.rabbitmq.com/
 .. _rabbitmq installation: http://docs.celeryproject.org/en/latest/getting-started/brokers/rabbitmq.html
+.. _rabbitmq website installation: http://www.rabbitmq.com/download.html
 .. _Celery installation: http://docs.celeryproject.org/en/latest/getting-started/introduction.html#bundles

docs/release-notes/release-notes-4.11.1.rst

@@ -0,0 +1,54 @@
+|RCE| 4.11.1 |RNS|
+------------------
+
+Release Date
+^^^^^^^^^^^^
+
+- 2018-02-02
+
+
+New Features
+^^^^^^^^^^^^
+
+
+
+General
+^^^^^^^
+
+
+
+Security
+^^^^^^^^
+
+
+
+Performance
+^^^^^^^^^^^
+
+
+
+Fixes
+^^^^^
+
+
+- Audit logs: handle query syntax errors in whoosh query parser. Prevents 500
+  errors on wrongly entered search terms in audit logs.
+
+- Mercurial: fix new 4.4.X code change that does strict requirement checks. Fixes
+  problems with Mercurial Largefiles repositories.
+
+- VCSServer: in case of errors in the VCSServer worker restart it gracefully.
+  In rare cases certain errors caused locking of workers, and
+  unresponsive connections. Now we restart a worker freeing up memory and
+  connection.
+
+- Git: handle flaky and slow connection issues with git. Due to the changes in
+  Pyramid, flaky connections started affecting git clones.
+
+
+Upgrade notes
+^^^^^^^^^^^^^
+
+- Fixed regression on git with high latency connections.
+  No upgrade problems should be expected, however please check GIT repos
+  behaviour on upgrade.

docs/release-notes/release-notes.rst

@@ -9,6 +9,7 @@ Release Notes
 .. toctree::
    :maxdepth: 1
 
+   release-notes-4.11.1.rst
    release-notes-4.11.0.rst
    release-notes-4.10.6.rst
    release-notes-4.10.5.rst

pkgs/python-packages.nix

@@ -1680,7 +1680,7 @@
     };
   };
   rhodecode-enterprise-ce = super.buildPythonPackage {
-    name = "rhodecode-enterprise-ce-4.11.0";
+    name = "rhodecode-enterprise-ce-4.11.1";
     buildInputs = with self; [pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage configobj];
     doCheck = true;
     propagatedBuildInputs = with self; [setuptools-scm amqp authomatic Babel Beaker celery Chameleon channelstream click colander configobj cssselect decorator deform docutils dogpile.cache dogpile.core ecdsa FormEncode future futures gnureadline infrae.cache iso8601 itsdangerous Jinja2 billiard kombu lxml Mako Markdown MarkupSafe msgpack-python MySQL-python objgraph packaging Paste PasteDeploy PasteScript pathlib2 peppercorn psutil psycopg2 py-bcrypt pycrypto pycurl pyflakes pygments-markdown-lexer Pygments pyparsing pyramid-beaker pyramid-debugtoolbar pyramid-jinja2 pyramid-mako pyramid pysqlite python-dateutil python-ldap python-memcached python-pam pytz pyzmq py-gfm recaptcha-client redis repoze.lru requests Routes setproctitle simplejson six SQLAlchemy sshpubkeys subprocess32 supervisor Tempita translationstring trollius urllib3 URLObject venusian WebError WebHelpers2 WebHelpers WebOb Whoosh wsgiref zope.cachedescriptors zope.deprecation zope.event zope.interface nbconvert bleach nbformat jupyter-client alembic invoke bumpversion transifex-client gevent greenlet gunicorn waitress uWSGI ipdb ipython CProfileV bottle rhodecode-tools appenlight-client pytest py pytest-cov pytest-sugar pytest-runner pytest-catchlog pytest-profiling gprof2dot pytest-timeout mock WebTest cov-core coverage];

rhodecode/i18n/rhodecode.pot

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: rhodecode-enterprise-ce 4.11.0\n"
 "Report-Msgid-Bugs-To: marcin@rhodecode.com\n"
-"POT-Creation-Date: 2018-01-30 17:32+0100\n"
+"POT-Creation-Date: 2018-01-30 18:06+0100\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -17,25 +17,6 @@ msgstr ""
 "Content-Transfer-Encoding: 8bit\n"
 "Generated-By: Babel 1.3\n"
 
-#: rhodecode/apps/_base/__init__.py:142
-#, python-format
-msgid "Error creating repository %s: invalid certificate"
-msgstr ""
-
-#: rhodecode/apps/_base/__init__.py:146
-#, python-format
-msgid "Error creating repository %s"
-msgstr ""
-
-#: rhodecode/apps/_base/__init__.py:219
-#, python-format
-msgid "The repository `%(repo_name)s` cannot be loaded in filesystem. Please check if it exist, or is not damaged."
-msgstr ""
-
-#: rhodecode/apps/_base/__init__.py:274
-msgid "Editing user `{}` is disabled."
-msgstr ""
-
 #: rhodecode/apps/admin/navigation.py:72 rhodecode/authentication/routes.py:60
 #: rhodecode/integrations/views.py:154
 #: rhodecode/templates/admin/permissions/permissions.mako:36

rhodecode/lib/middleware/simplevcs.py

@@ -248,8 +248,11 @@ class SimpleVCS(object):
                 repo_name, db_repo.repo_type, scm_type)
             return False
 
-        return is_valid_repo(repo_name, base_path,
-                             explicit_scm=scm_type, expect_scm=scm_type)
+        config = db_repo._config
+        config.set('extensions', 'largefiles', '')
+        return is_valid_repo(
+            repo_name, base_path,
+            explicit_scm=scm_type, expect_scm=scm_type, config=config)
 
     def valid_and_active_user(self, user):
         """

rhodecode/lib/user_log_filter.py

@@ -106,11 +106,14 @@ def user_log_filter(user_log, search_term):
             elif isinstance(term, query.DateRange):
                 field = term.fieldname
                 val = [term.startdate, term.enddate]
+            elif isinstance(term, query.NullQuery.__class__):
+                field = ''
+                val = ''
             else:
                 field = term.fieldname
                 val = term.text
-
-            user_log = user_log.filter(get_filterion(field, val, term))
+            if field:
+                user_log = user_log.filter(get_filterion(field, val, term))
     elif isinstance(qry, query.Or):
         filters = []
         for term in qry:

rhodecode/lib/utils.py

@@ -240,7 +240,7 @@ def _is_dir_writable(path):
     return True
 
 
-def is_valid_repo(repo_name, base_path, expect_scm=None, explicit_scm=None):
+def is_valid_repo(repo_name, base_path, expect_scm=None, explicit_scm=None, config=None):
     """
     Returns True if given path is a valid repository False otherwise.
     If expect_scm param is given also, compare if given scm is the same
@@ -251,6 +251,7 @@ def is_valid_repo(repo_name, base_path, expect_scm=None, explicit_scm=None):
     :param base_path:
     :param expect_scm:
     :param explicit_scm:
+    :param config:
 
     :return True: if given path is a valid repository
     """
@@ -260,7 +261,8 @@ def is_valid_repo(repo_name, base_path, expect_scm=None, explicit_scm=None):
 
     try:
         if explicit_scm:
-            detected_scms = [get_scm_backend(explicit_scm)(full_path).alias]
+            detected_scms = [get_scm_backend(explicit_scm)(
+                full_path, config=config).alias]
         else:
             detected_scms = get_scm(full_path)
 

rhodecode/lib/vcs/backends/base.py

@@ -203,6 +203,14 @@ class BaseRepository(object):
     def __ne__(self, other):
         return not self.__eq__(other)
 
+    @classmethod
+    def get_default_config(cls, default=None):
+        config = Config()
+        if default and isinstance(default, list):
+            for section, key, val in default:
+                config.set(section, key, val)
+        return config
+
     @LazyProperty
     def EMPTY_COMMIT(self):
         return EmptyCommit(self.EMPTY_COMMIT_ID)

rhodecode/lib/vcs/backends/git/repository.py

@@ -62,7 +62,7 @@ class GitRepository(BaseRepository):
                  update_after_clone=False, with_wire=None, bare=False):
 
         self.path = safe_str(os.path.abspath(repo_path))
-        self.config = config if config else Config()
+        self.config = config if config else self.get_default_config()
         self._remote = connection.Git(
             self.path, self.config, with_wire=with_wire)
 

rhodecode/lib/vcs/backends/hg/repository.py

@@ -72,8 +72,14 @@ class MercurialRepository(BaseRepository):
         :param update_after_clone=False: sets update of working copy after
            making a clone
         """
+
         self.path = safe_str(os.path.abspath(repo_path))
-        self.config = config if config else Config()
+        # mercurial since 4.4.X requires certain configuration to be present
+        # because sometimes we init the repos with config we need to meet
+        # special requirements
+        self.config = config if config else self.get_default_config(
+            default=[('extensions', 'largefiles', '1')])
+
         self._remote = connection.Hg(
             self.path, self.config, with_wire=with_wire)
 

rhodecode/lib/vcs/backends/svn/repository.py

@@ -71,7 +71,7 @@ class SubversionRepository(base.BaseRepository):
     def __init__(self, repo_path, config=None, create=False, src_url=None,
                  **kwargs):
         self.path = safe_str(os.path.abspath(repo_path))
-        self.config = config if config else base.Config()
+        self.config = config if config else self.get_default_config()
         self._remote = connection.Svn(
             self.path, self.config)
 

rhodecode/model/db.py

@@ -1760,18 +1760,6 @@ class Repository(Base, BaseModel):
         q = q.options(FromCache("sql_cache_short", "repository_repo_path"))
         return q.one().ui_value
 
-    @classmethod
-    def is_valid(cls, repo_name):
-        """
-        returns True if given repo name is a valid filesystem repository
-
-        :param cls:
-        :param repo_name:
-        """
-        from rhodecode.lib.utils import is_valid_repo
-
-        return is_valid_repo(repo_name, cls.base_path())
-
     @classmethod
     def get_all_repos(cls, user_id=Optional(None), group_id=Optional(None),
                       case_insensitive=True):